The CORE Rankings
Some may know that a few years ago, the Australian Research Council (ARC) had a ranking of publication outlets produced. For computer science, the exercise was outsourced to CORE, the association of Australian and NZ CS departments (Oz equivalent of CRA). It categorised conferences (and journals) into A*, A, B and C venues.
I have in the past stated what I think of that list: very little. In short, I think it’s highly compromised and an embarrassment for Australian computer science. And I’m outright appalled when I see that other countries are adopting the “CORE Rankings”!
The ARC disendorsed the rankings in 2011. Yet, in 2013, CORE decided to maintain and update it. I argued that updating with a similar process as the original one will not improve the list.
The Fellows Letter
Now, some senior colleagues (Fellows of the Australian Academy of Science) have written an open letter, denouncing not only the CORE list, but basically any use of publication venues as an indicator of research quality.
The letter was, apparently, written by Prof Bob Williamson from the ANU, and fellow group leader at NICTA. Bob is a guy I have a lot of respect for, and we rarely disagree. Here we do completely. I also highly respect the other Fellows (one of them is my boss).
The Fellows essentially argue (with more clarification by Bob) that looking at where a person has published is useless, and the right way to judge a researcher’s work is to read their papers.
What I think
With all respect, I think this is just plain nonsense:
These rankings exist, like it or not. In fact, we all use them all the time. (Ok, I cannot prove that the “all” bit is strictly true, some, like Bob, may not, the rest of us do.) When I look at a CV, the first thing I look for is where did they publish. And I claim that is what most people do. And I claim it makes sense.
Fact is that we know what the “good” venues are in our respective disciplines. This is where we send our papers to, this is where we tell our students and ECRs they need to get their papers accepted. They are the yardsticks of the community, like it or not, it is where you publish to have impact. Publishing in the right venues leads to high citations, publishing in the wrong ones doesn’t.
Of course, we really only understand the venues in our own sub-disciplines, and may be a few neighbouring ones. So, collecting and documenting these top venues across all of CS isn’t a bad thing, it creates clarity.
The idea that someone can judge a person’s work simply by reading some of their papers (even the self-selected best ones), with respect, borders on arrogance. In effect, what this is saying is that someone from a different sub-discipline can judge what is good/significant/relevant work!
If this was true, then we as a community could reduce our workload drastically: We’d stop having conference PCs where everyone has to read 30 papers, and every paper gets at least half a dozen reviews before being accepted (as at OSDI, where I’m presently struggling to get all my reviews done). Instead, every conference would simply convene a handful of Bobs, divide the submissions between them, and each decides which one of their share of the papers should be accepted.
Of course, things don’t work like this, for good reasons. I’ve served on enough top-tier conference PCs to have experienced plenty of cases where the reviews of discipline experts diverge drastically on multiple papers. In my present OSDI stack of 29 papers this is true for about 35% of papers: 10 papers have at least one clear reject and one clear accept! And it is the reason why each paper gets at least 6 reviews: we get the full spectrum, and then at the PC meeting work out who’s right and who’s wrong. The result is still imperfect, but vastly superior to relying on a simple opinion.
Now these reviewers are the discipline experts (in this case, leading researchers in “systems”, incorporating mostly operating systems and distributed systems). If you get such a diversity of opinions within such a relatively narrow subdiscipline, how much would you get across all of computer science? I certainly would not claim to be able to judge the quality of a paper in 80% of computer science, and someone thinks they can, then my respect for them is taking a serious hit.
In summary, I think the idea that someone, even one of the brightest computer scientists, can judge an arbitrary CS paper for its significance is simply indefensible. An expert PC of a top conference accepting a paper has far more significance than the opinion of a discipline outsider, even a bright one!
Of course, that doesn’t justify using the publication outlets as the only criterion for promotion/hiring or anything else. That’s why we do interviews, request letters etc. Also, I definitely believe that citations are a better metric (still imperfect). But citations are a useless measure for fresh PhDs, and mostly of not much use for ECRs.
Nor do I want to defend the present CORE list in any way. I said that before, but I’m repeating for completeness: the present CORE list is the result of an utterly broken process, is completely compromised, and an embarrassment for Australian computer science. And any attempt to fix it by using the existing process (or some minor variant of it) is not going to fix this. The list must either be abandoned or re-done from scratch, using a sound, robust and transparent process.
My arguments only are about top venues. A track record of publishing in those means something, and identifying across all of CS what those top venues are has a value. By the same token I believe trying to categorise further (i.e. B- and C-grade venues, as done in the CORE list) is a complete waste of time. Publishing in such venues means nothing (other than positively establishing that someone has low standards). So, if we bother to have a list, it should only be a list of discipline top venues, nothing more.
Note that as far as the newspaper article reported on facts it got them essentially right. However, it drew the wrong conclusions, based on an incorrect understanding of the situation and the realities of commercialisation, and these incorrect conclusions triggered Nick’s blog. My previous blog addressed issues around commercialisation of ICT IP. Now I’d like to address some of the specifics of the OK Labs situation, and the NICTA IP involved.
Before delving deeper, I must say that there are severe limits to what I can reveal. I was a director of OK Labs, and as such bound by law to confidentiality with regard to information which I obtained as a director. Beyond that there are confidentiality clauses affecting the main shareholders (which includes myself as well as NICTA). I also was an employee of OK Labs from early 2007 until mid 2010. Essentially I have to restrict my comments to what’s on the public record or was known to me before the respective agreements were signed.
A tale of three microkernels and four names
First there is the issue of the three kernels, the first one appearing under two names, which continues to create confusion, even though the facts were essentially correctly reported in the newspaper article.
Before OK Labs there was NICTA’s version of the L4 microkernel. This was an evolution of the open-source Pistachio microkernel, originally mostly developed at Karlsruhe University in Germany. We had ported it to a number of architectures, including ARM, had optimised it for use in resource-constrained embedded systems, and had designed and implemented some really cool way of doing context switches really fast (factor 50 faster than Linux). We had also ported Linux to run on top (i.e. used L4 as a hypervisor to support a virtualised Linux). Thanks to the fast context-switching technology, that virtualized Linux ran faster than native.
As I said, this microkernel started off as open-source (BSD license), and remained open-source. While the BSD license would have allowed us to fork a closed-source version (while acknowledging the original authors) this would have been a stupid thing to do. We wanted our research outcomes to be used as widely as possible.
In 2004, that L4 microkernel caught the attention of Qualcomm. They had two specific (and quite different) technical problems for which they were looking for solutions. One required a fast, real-time capable kernel with memory protection. The other required virtualization of Linux on ARM. Our L4 provided both, and nothing else out there came close.
Qualcomm engaged NICTA in a consulting arrangement to help them deploy L4 on their wireless communication chips. The initial evaluations and prototyping went well, and they decided to use L4 as the basis of their firmware.
This was all before OK Labs was founded. In fact, at the time we created OK Labs, the first phones with L4 inside were already shipping in Japan! And all based on the open-source kernel.
The engagement with Qualcomm grew to a volume where it was too significant a development/engineering effort to be done inside the research organisation. In fact, the consulting revenue started to threaten NICTA’s tax-free status! Furthermore, we saw a commercial opportunity which required taking business risks, something you can’t do with taxpayer $$. This is why we decided to spin the activity out as OK Labs. OK Labs marketed L4 under the name “OKL4 microkernel”, and continued its development into a commercial-grade platform.
OK Labs initially operated as a services business, serving Qualcomm, but also other customers. Note that they didn’t even need NICTA’s permission to do this, they took an open-source release and supported it. Anyone could have done this (but, of course, the people who had created the technology in the first place were best placed for it). Among others, this meant that there was never any question of royalties to NICTA.
Also, it is important to note that Qualcomm would almost certainly not have adopted L4 if it wasn’t open source. Their style is to do stuff in-house, and it would have been their natural approach to just re-do L4. The engagement with us was unusual for them, but it led to NICTA technology being deployed in over 1.5 billion devices.
OK Labs later decided to become a product company, and seek VC investment to enable this. They developed their own product, the OKL4 Microvisor. This is the successor of the OKL4 microkernel, and was developed by OK Labs from scratch, NICTA (or anyone else) has no claim to it. It is licensed (and is shipping) on a royalty basis, which is exactly what you expect from a product company.
Then there is the third microkernel, seL4. This was developed from scratch by NICTA, and its implementation mathematically proved correct with respect to a specification.
The correctness proof was the big-news event that made headlines around the world. It is truly groundbreaking, but primarily as a scientific achievement: something people had tried since the ’70s and later put into the too-hard basket. But, as per my atomic-bomb metaphor in the previous blog, once people know it’s possible they can figure out how to do it themselves. Particularly since we had published the basics of the approach (after all, doing research is NICTA’s prime job, and it’s not research if it isn’t published). And it’s seL4’s development (and all the stuff that made its verification possible) that took 25 person years. This is the effort behind the biggest ICT research success that came out of Australia in a long time. It’s fair to say that this has put NICTA on the map internationally.
seL4 is nevertheless something that can be turned into an exciting product, but that needs work. As argued in the previous blog, that’s not something you do in a research lab, it’s company business. That’s why NICTA needed a commercialisation channel.
The way they decided to do it was to license seL4 exclusively to OK Labs, with a buy-out option (i.e. the option to acquire the IP outright) on achieving certain milestones (for the reasons explained in the previous blog). In exchange, NICTA took equity (i.e. a shareholding) in OK Labs, as a way to get returns back if commercialisation succeeded. Using OK Labs as the commercialisation vehicle was an obvious choice: Firstly, OK Labs was developing the market and distribution for this kind of technology. Secondly, OK Labs does all its engineering in Australia, and any alternative would have been overseas. A reasonable deal.
How about national benefit?
The (more or less clearly stated) implication from the commentators that NICTA made a mistake is totally unfounded. And that should not come as a surprise: the people involved in the decision knew what they were doing. The director of the NICTA Lab where the work happened was Dr Terry Percival. He happens to be the person whose name is on the much-lauded CSIRO wifi patent! And NICTA’s CEO at the time was Dr David Skellern. He was the co-founder or Radiata, which implemented CSIRO’s wifi invention in hardware, and got sold for big bucks to CISCO! Those guys knew a bit about how to commercialise IP!
There are comments about the “non-discussion of how much money changed hands”. Well, that happens to be part of the stuff I can’t talk about, for the reasons listed at the beginning.
Also, national benefits aren’t simply measured in money returned to NICTA. There are other benefits here. For one, there is a publicly stated intent by OK Labs’ purchaser, General Dynamics (GD), to not only maintain but actually expand the engineering operation in Australia. Also, one thing we learn is that technology like seL4 isn’t trivial to commercialise, it requires a big investment. GD has the resources to do this, and is active in the right markets, so has the distribution channels. Finally, there is a lot of on-ging research in NICTA which builds on seL4, and is building other pieces which will be required to make best use of seL4. NICTA owns all this, and is certain to stay in the loop. Furthermore, we now have a lot of credibility in the high-security and safety-critical space. This has already shown tangible outcomes, some of which will be announced in the next few weeks.
Did we make all the best decisions? This is hard to say even with the benefit of hindsight. We certainly made good decisions based on the data at hand. The only sensible alternative (both then and with the benefit of hindsight) would have been to open-source seL4, as we had done with L4 earlier. This might or might not have been the best for maximising our long-term impact and national benefit.
We can’t really tell, but what we do know is that we’re in a very strong position of being able to do more high-impact research. In fact, we have already changed the way people think about security/safety-critical systems, and we are likely to completely change they way future such systems will be designed, implemented and verified.