Skip to content

OK Labs Story (9): The End Game

My October ’09 report to the Board was basically ignored. The more critical one of the investors was initially interested in following up, but in the end nothing happened. Steve obviously worked on him, but we also got, at a perfectly inopportune time, two deals which seemed to prove me wrong.

The AMSS disaster

One had started earlier, an Asian phone manufacturer wanting to build a “mass-market smartphone”, i.e. Android and modem stack on the same low-end (ARM9) processor core. I never understood why, and I was kept away from the customer or any real information from them, so I can only suspect that they were after an iPhone lookalike on the cheap (where looks would be more important than functionality).

So, we actually build a new product specifically for outdated hardware (the successor core ARM11, an ARMv6 implementation, was already in widespread use then). Technically it was a fun challenge. The project, code-named Marvin (the paranoid Android, paranoid because he’s never run on an ARM9 before) back-ported Android to ARMv5. The interesting bit was to get performance on the processor, which features a virtually-indexed, virtually-tagged cache. Windows as well as Linux deal with this cache by flushing it on every context switch, a performance killer. However, we had many years ago invented an approach to fast context switching on ARMv4/5, which made Linux faster by virtualising it; we used the same approach on the original virtualised phone, the Motorola Evoke. So we could run Android on that processor pretty much as fast as it could possibly be, and our engineering team made it work, of course.

The catch was that the customer was using Qualcomm chips, and was therefore running Qualcomm’s AMSS modem stack. So we had to port AMSS to the Microvisor. Technically this was not a big problem for our engineering team (and they did it just fine), but there was a legal issue: we did not have a source license to AMSS. The customer claimed that they had the rights to sub-license for the purpose. I voiced concerns at the board, resulting in a supposedly thorough analysis of the legal situation, which supposedly showed it was allright.

Obviously, the clean thing would have been to simply talk to Qualcomm, as they were bound to find out eventually. But for some reason Steve thought he could get away without telling them. (This is consistent with Steve’s 90-day horizon, he routinely goes for short-term gains even if this seriously damages long-term prospects, while my attitude is exactly the opposite.) It beats me why the rest of the Board followed.

Of course, Qualcomm eventually found out, and sent us a cease-and-desist letter, with which we complied. That (which happened after the presentation of my market analysis) was the end of this deal, but also of our once-great relationship with Qualcomm.

The poisonous Dash deal

There was another project which seemed to undermine my analysis, the only one that ever came through one of our investors. One of the directors had long-running high-level links with a major US network operator, let’s call them “Dash Mobile”. He got them to contract us for a prototype “mass-market smartphone”, running Android on an ARM9-based, single-core phone provided by another Asian manufacturer (not involving Qualcomm IP). Again, I never got enough information to understand why Dash was interested in this and whether their reasons added up. But, given the small size of the contract, it might be them just using some spare cash to investigate something which could be seen as having the potential for some market disruption, which Dash urgently needed (they weren’t going so great at the time).

Needless to say, the project went no-where (although that didn’t become obvious until after my departure). OK engineering delivered, as usual, and our overheads were low enough to be in the noise margin. But running a smartphone OS on the low-end processor did simply not result in a satisfying user experience (even with zero virtualisation overhead). But for a long time it seemed like a great project totally in line with Steve’s PR, contrary to what I was saying, and thus undermined my credibility. By the time I was proven right, it was too late. The timing of the Dash deal was disastrous.

In the end, the Marvin project was damaging in several ways. On the one hand we wasted a year or so, and millions of dollars, on developing a product for which there was no market, it was a complete waste. On the other hand it was a serious hit on morale in the engineering team: people had worked their asses off for many months, including regular work on week-ends, to meet the schedule. And in the end they saw that, while the work was technically good, the resulting product was useless. Several of the top people left after that.

Blood in the Board Room

Eventually I realised that I was powerless to do anything constructive in the company. Furthermore, 90% of frustration in my live originated from OK – in the mornings I dreaded looking at my email, which was full of shit having come in from Chicago overnight. At the same time, most of my fun came from research and working with bright students, and I was doing less and less of it. I decided I was too old to play such a masochist game and had to get out.

However, I thought I owed it to the many great OK engineers to at least make one more attempt to change things, unlikely as it was to succeed. So, in May 2010 I forced a bloodbath in the boardroom, and, predictably, the blood was mine, as the investors backed the CEO. Note that to this date we had a single quarter where company revenue and booking performance had matched the goals, and that was only because a large Motorola deal had come in late and spilled into a quarter that had low expectations; the sum of the two quarters was still well below expectations. But, of course, salvation was just around the corner, and we would soon be profitable. I have no idea why the VCs kept buying this…

The exit

I was given a face-saving 12-month consulting deal for leaving at the end of June ’10, and remained on the Board.

Incidentally, Benno left at the same time, voluntarily. The fundamental reason was the same: he couldn’t stand the bullshit and daily frustration any longer. (Benno and I are great friends again, I was invited to his wedding the following year, and his new startup Breakaway Consulting is a close partner of NICTA. We’re jointly developing eChronos, a verified unprotected RTOS which is already being licensed for medical implants. And he founded Cog Systems, the moral successor of OK Labs.)

The internal announcement of the departures was telling. I was asked to draft my own “to ensure it’s correct”. I kept it brief and focussed on simple, public facts, and, of course, didn’t praise myself, that would be the job of others. Steve mailed it out as is, with no attempt to mention my contributions to the company. In contrast, what Steve wrote about Benno was full of praise.

Nevertheless, were also both asked to pack up our desks after hours to avoid “unsettling” people. Steve has always under-estimated the intelligence of our engineers, and generally has no clue how they tick. People were already fairly unsettled, and the way I was farewelled clearly looked like a sacking to everyone. Given that the majority was there because of me, this obviously didn’t help morale at all.

While following Steve’s instructions to the letter, I did my best to counteract the effects on morale in engineering. I met up with them for lunch, putting up a cheerful face and taking positively about the whole development. Not sure how much this helped in the end, but it would certainly have been better than just vanishing the way Steve wanted me to.

The sale

With Benno and myself gone, the company had no-one left with a technical vision. All they could do from now on was implementing the designs Benno had made before leaving, and do whatever Chicago dreamt up. Needless to say, the company’s performance didn’t improve. The investors lost patience and wanted an exit: OK Labs was put up for sale.

The search for a buyer dragged on and on. Hardly surprising, of course, given that the “business plan” was a joke that didn’t hold up to scrutiny by anyone with technical insight. The “business plan” was built around our “enterprise strategy”, which basically meant “we’ll sell to enterprises” without any clue why they would buy from us, there was no value-add anyone could formulate in a way I could understand. Any real opportunities were outside that “strategy”.

Ironically, although maybe not surprisingly, by the time the company was finally sold, the only royalty deals we ever made were exactly in the two areas I had championed: automotive (where we may be in cars by now) and security (where there is at least one product in use by the military of a NATO country, probably more). But never a cent in royalties in consumer/enterprise mobile!

The strategy for the sale could only be to find someone interested in buying the engineering team (no longer the A team, a few individuals notwithstanding, but still very good) or the existing customer relationships.

What amazed me (despite all I had seen in the past) was the slide deck which was used to shop the company around. As part of the second investment round, OK Labs had acquired all rights to seL4 and its verification, and NICTA had delivered it sometime in 2010. Yet the sales deck didn’t mention seL4 a single time! The only obscure reference was the highly-misleading bullet point “World’s only fully verified Microvisor” (it’s seL4 that was verified, not the OKL4 “Microvisor”).

This is truly stunning! In 2011, MIT Technology Review listed seL4 in its “TR10″, what it judges the ten most important breakthrough technologies. So, OK sat on this truly unique technology, and the CEO had no clue what to do with it! Steve simply never understood seL4 and its potential. Which must make OK Labs the only company ever that owned a TR-10 technology but didn’t know what to do with it!

The way he came to his assessment that seL4 wasn’t of any use was a “market validation” exercise we had done earlier. It used the process Rob had used in the past at Wind River: You’ve got some ideas on how to evolve your product, so you go to all your customers and ask them what they thought about it. Which is a perfectly fine approach for a company with an established products it wants to improve, and an established, large customer base. In our case (where we had no relevant customers) we went to potential customers, all established players in their field and, effectively, asked them “hey, we’ve got this great technology that will completely disrupt your business, are you interested?” Can anyone guess the answer? But this is how strategy was decided at OK Labs.

The spoils

In the end, the company was sold to General Dynamics in August 2012, for a price that gave our investors more or less their money back.  None of the common shareholders saw a cent. In the dying days of the company, the investors had doubled as loan sharks, bridging the company at incredible interest rates, making double sure that there was nothing left for anyone else. (Actually, I was offered $1000 for signing away some irrelevant rights, which I declined, as being below my dignity.)

The only person who made a profit was the one who comprehensively fucked it up. Steve got about $900k in a combination of completion and retention bonuses (and on top of that may have retained the full-year separation pay he had written into his own employment contract). Isn’t the world of business wonderful?


© 2014 by Gernot Heiser. All rights reserved.

Permission granted for verbatim reproduction, provided the reproduction is of the complete, unmodified text, is not made for commercial gain, and this copyright note is included in full. Fair-use abstracting permitted.

StartPrevious … Next

OK Labs Story (8): Competitors and Markets

Competitors

Early on we found that we were competing with two other players: Jaluna, later renamed VirtualLogix, and Trango, both from France (although Jaluna was headquartered in California).

Jaluna/VirtualLogix

Jaluna goes back to the early microkernel days: INRIA-spinout Chorus Systèmes, who prided themselves of having the first commercial microkernel (in the 1980s). Like other microkernels of the first generation, its performance sucked and it was practically unusable. They early on virtualised Unix on Chorus, but due to poor performance moved the Unix kernel into the Chorus kernel (like these days BSD is co-located with Mach in the Mac OS X kernel), which is no-longer virtualisation. Liedtke’s original L4 (in 1993) out-performed Chorus by a factor of ten!

Chorus had some deployments in the network-infrastructure market, and were eventually bought by Sun, but later closed down, and bought out by the original founders as Jaluna. 20-30 years after the original Chorus, their technology was still essentially Chorus, they don’t seem to have learnt much. They were “virtualising” Linux now, still by putting it into the kernel, something I call pseudo-virtualisation. In an earlier blog I joked about them taking my Advanced OS course so they could learn how to build performant microkernels… At some time they changed their name to VirtualLogix.

Their most noteworthy “achievement” was to demonstrate their loserdom with an incredibly shoddy paper published in the IEEE CCNC conference. They “showed” that their performance was way better than ours, which I knew not to be true (plenty of potential customers did their own evaluation showing this). I had fun in a series of blogs ripping that paper to pieces, and students in my Advanced OS course did similar. That paper was so full of fairly obvious defects, it should never have passed peer review. Needless to say, the conference is on my blacklist.

What almost certainly happened with their “performance” measurements is that they took our open-source version as is and built it blindly. The release had debugging and profiling enabled by default, which, of course is a performance killer. Any ethical comparative benchmarking would have disabled that.

VirtualLogix were later bought by Israeli device-management-software provider RedBend.

Trango

The other was Trango Virtual Processors, founded by a bunch of ex-ST Microelectronics employees. Trango was a hypervisor for ARM (and MIPS) processors. Their performance was much better, but bought with a 100% assembler implementation! This is good for performance, but implies huge costs for maintenance and adapting to changing conditions. And, while their founders might have been good hackers, they weren’t technology leaders the way we were.

Trango was later bought by VMware. I never understood why, and I don’t think VMware did. In fact, I believe they soon realised they bought a lemon. They marketed the Trango technology for a while as MVP (mobile virtualisation platform) but then went quiet, and about a year later came out with a new product under the same name, which shared nothing (probably not a line of code) with the original. It was essentially turning the Android Linux kernel into a hypervisor (KVM-style), to support running a second OS in a virtual machine. This was targeted to the “BYOD” business market, where enterprises would install a VM with the business (logical) phone on an employee’s handset. The problem is that this adds very little to security over just using the vanilla phone for accessing the enterprise IT infrastructure: As the hypervisor is the Android OS of the private (logical) phone, compromising it will automatically compromise the VM in which the business phone runs. They used encryption of data at rest to mitigate this, but that’s not really much more than window dressing.

Markets

Neither VirtualLogix nor Trango ever were a real threat: We never lost a deal we were directly competing with either of them (they may have won some we didn’t know about).

Mobile

And we had many opportunities. Basically none were created by our (overpaid) sales force (except for Motorola deals generated by Tony/Josh). Instead, people were approaching us, from across a range of industry sectors. Some of them, especially automotive, appeared to me as great opportunities from the beginning. Of course, a startup has to focus, and Steve decided to focus on mobile. Why? There were no technical reasons, no real analysis of our value add, it was a simple calculation of >1 billion devices sold worldwide each year, if we can get on 20% of them and get $1 per unit, we’re rich. There was never any justification of why these figures were realistic, but Steve told the investors what they wanted to hear, and they lacked the insight to realise it was all bullshit.

For a while almost the whole marketing of our competitor VirtualLogix was directed at “single-core feature phones”, i.e. running a simple application OS and the modem stack on the same core. It was obvious that this would be, if at all, a very short-lived niche, given that the incremental cost of an additional core trended towards negative. In fact, I joked internally that if I was them and it’s all I had, I’d be very worried. Imagine my dismay when, about a year later, our marketing homed in on exactly that use case! Except we called it “mass-market smartphone”. The “mass-market” bit was to refer to the simple ARM9-based hardware of the then feature-phones, and the “smartphone” on the ability to run a smartphone OS (Android).

The thought that smartphone apps (even ignoring OS overheads) require grunty processors, and that it made no sense to share those with the baseband, wasn’t understood in Chicago (and, as I mentioned earlier, critical thinking was treason).

In this context I did a study of industry sectors, the role virtualisation could play in them, and attempted to quantify the value-add. I presented this at a phone conference of board members in October 2009. The conclusions I drew was that I could not see a significant value-add for our technology in mobile, no killer apps, and a low bar for entry for competitors, resulting in very low margins. In addition, VMware had it in their hands to reduce these margins to zero: royalties from the mobile hypervisor could hardly be central to their business model, so they could afford giving it away for free, and thus eliminating our margins. And soon a new competitor appeared: an approach from Columbia University to virtualise Android at the OS ABI level, which has the advantage of simplicity (the paper was published at SOSP in October ’11 and I had in fact reviewed and shepherded it, so I knew all about it). They created a startup going after the same market.

In my analysis, I concluded that I could not see the total addressable market exceeding $100M/a in this space, and that would be shared between many competitors with shrinking margins. Not an attractive place to be.

Instead my analysis identified two promising areas: automotive and strong security.

Automotive

At the time we actually had traction in automotive: several component manufacturers had approached us. Despite Steve’s best attempts at chasing them away, some were highly persistent. I was particularly excited about one potential partner: OpenSynergy, a startup in Germany that came out of the research lab of a major car manufacturer. They had approached us and I had visited them as far back as July ’07. They started developing an automotive virtualisation product on top of our open-source microkernel and tried to negotiate a partnership agreement with us. Steve’s idea of “partnership”, however, was indistinguishable from “customer”: you pay and we deliver something to you (and that after long and painful negotiations Steve-style). They were perfect as a real partner: they had all the domain knowledge as well as excellent networks in the industry, both of which we were lacking.

Steve stuffed them around for two full years until they finally decided to go with someone else: German Sysgo (now bought by Thales) who had an aerospace (DO178-B) certified L4 clone named PikeOS. They would hardly have been happy with that marriage: I knew that PikeOS’s performance was miles away from ours, and we also knew from other engagements how performance-sensitive the automotive use cases were. Nevertheless, we managed to create a competitor out of nothing in a space we could (and should) have owned! Clearly this takes a special kind of talent.

Actually, in my “bullshit” mail folder I still have the mail Steve sent around OK Labs when OpenSynergy’s partnership with Sysgo was announced. He mailed around their press release with the subject line “1 Loser + 1 Loser makes for the Biggest Loser!” The conclusion was correct (although not as intended): the Biggest Loser was us, and that was immediately clear to me.

Our other engagement in the automotive space was with a company called ADIT, which is a joint venture of the two biggest automotive suppliers Bosch (Germany) and Denzo (Japan). Their German arm was developing integrated infotainment head units for Bosch, and they needed virtualisation to run automotive real-time components (with very strict real-time and performance requirements) concurrently with Linux-based infotainment stuff. Again, OK head office stuffed them around, apparently trying to make them feel that we’d be doing them a huge favour by engaging. Even when we had a development contract, they were not taken seriously, and insufficient resources allocated.

I still vividly remember one of the few customer meetings I had at that time. I was in Germany for CeBIT representing NICTA, and Abi thought I could really help rescuing the ADIT project, so I joined him for a half-day meeting in Hildesheim. The meeting superficially went fine, but during a break, ADIT’s Chief Purchaser took me aside and gave me (in German) the most serious dress-down I had experienced in my business career, calling OK Labs “the most unreliable outfit he’s ever dealt with”. This hurt deeply, because on the one hand I could see he was right, but on the other hand I knew that Abi (the technical sales guy whose project it was) as well as our engineers were doing their very best. In fact, ADIT engineers held ours in high regard (a recurring pattern). It was just head office fucking them around endlessly. Fortunately, things improved a lot after that meeting, and the project was, in the end, completed on schedule and to spec, and ADIT was happy with the outcome.

In automotive I could clearly see the value-add, and, while total numbers were less than in mobile, the addressable market looks way bigger. (Also, there’s a place for more than one hypervisor per car, which changes numbers significantly.) So I recommended making this area a main focus (despite having already missed the chance to own this market outright).

Security

The second attractive space was strong security. Not the soft notion of “security” enterprises seem to be content with, where there are lots of simple “good-enough” approaches, such as VMware’s MVP, but the more paranoid groups like the national-security sector. This is where strong isolation, as we provided it, would matter. A typical use case (but there are others) would be to turn a more-or-less COTS phone into a secure communication device, a secure (logical) phone running on the same hardware as a soldier’s or emergency responder’s personal phone.

While this is a market with established suppliers (Green Hills, Wind River, LynuxWorks) I knew from what I could gather about their technology (as well as learning about evaluations from customers) that their hypervisors were unable to deliver the performance required, whereas we could. Obviously, in this space the number of sold units is much smaller than automotive (leave alone consumer mobile), however margins are much bigger. And we also had a number of exploratory projects, so there was clearly interest. Also, this was the space for which NICTA’s seL4 was made and would give us an unmatchable advantage, and a close collaboration with NICTA was core to my strategy.

Hence, my conclusions for the OK strategy were clear: avoid being trapped in a market with dubious value-add and shrinking margins (consumer mobile) and focus on the two domains where I could see an addressable market of significant size and we had significant competitive advantage: automotive and security.


© 2014 by Gernot Heiser. All rights reserved.

Permission granted for verbatim reproduction, provided the reproduction is of the complete, unmodified text, is not made for commercial gain, and this copyright note is included in full. Fair-use abstracting permitted.

StartPreviousNext

OK Labs Story (7): The Investors

We had a total of three investors: In the first round we got an Australian VC. In the second round we got a US-based VC, plus Citrix as a strategic investor. Citrix only had observer status on the Board.

Fundamentally, the Board was weak.

One thing I learned is that Australian VCs have severe limits. Australia is a small pool, and VCs have to be very broad, which means they lack the technical expertise to really understand what they are investing in. In contrast, the top-tier Silicon Valley (“Sandhill Road”) VCs operate in a huge market, and specialise on particular segments, where they build up considerable expertise. Also, they have their Stanford and Berkeley professors at hand for in-depth technical assessments.

I saw this at play when we tried to raise a B round. None of the top-tiers wanted to bite, for obvious reasons: Our business plan was a combination of bullshit and wishful thinking. We had the best technology, but no inherent reason why it couldn’t be replicated (in fact, I knew I could take a half-dozen of the latest student generation and do it). And we had no convincing story of the value-add we were providing in the area we were targeting.

So, we ended up with a second- or third-tier VC, and it wasn’t a B round either, but a re-opening of the A round.

A top-notch VC not only brings in money, but also market insights and connections leading to deals. We didn’t get any of that from our VCs.

Steve had one of our directors eating out of his hand from early on, to the very end that person thought Steve was the greatest thing since bottled beer. He was a total walk-over for Steve. The other one was more critical, but didn’t really add much either. He probably noticed that there was a lot of BS, but couldn’t really dig deeper to keep the CEO honest. Which is actually quite an achievement by Steve, that they kept backing him despite never delivering on any of the sales or revenue targets. But the breakthrough was always around the corner, with all those great deals about to happen, and we were really just a couple quarters away from being cash positive…

The only deal our VCs ever got us was the fatal Dash one, but that’s for a later section. In the end, they were no more effective in holding Steve to account than I was, and they didn’t help develop the business at all.

Citrix should have been a chance, but they were restricted to observer status, and therefore couldn’t do much. Martin, the head of the Citrix CTO office who was the Board observer, saw through a lot of the bullshit, but wasn’t empowered to do much. Maybe it would have helped if I had built a closer relationship to him earlier.


© 2014 by Gernot Heiser. All rights reserved.

Permission granted for verbatim reproduction, provided the reproduction is of the complete, unmodified text, is not made for commercial gain, and this copyright note is included in full. Fair-use abstracting permitted.

StartPreviousNext

OK Labs Story (6): The Management Team

Steve is an incredible networker (which helps collecting intelligence, he’s got an excellent machine for picking up market rumours). His network also includes a lot of people whom he can draw on as consultants or employees.

Early in the life of the company he moved back to Chicago, to set up the company headquarters there. We needed a US presence (and parent company) and Steve initially ran it from his home. He soon got an excellent deal for office space in the heart of the Chicago CBD, exactly opposite the Willis Tower. Chicago isn’t the most obvious place, and time-zone wise, the West Coast would have been better, but in the end I don’t think it mattered. Chicago is well-connected and Motorola’s mobile business (who should have been a major customer) was there.

He also went on to hire an executive team: CFO, VP Product Management, VP Marketing, all based in Chicago. The VP Sales (eventually also Chicago-based) took years, including some false starts: one resigned within 2 weeks for family reasons, and a French guy resigned after the first phone meeting (never met either, but maybe they sensed more issues than I thought?) Together with Benno and myself this comprised the initial executive team.

Generally the executives Steve hired in the first round were nice and decent folks, I liked working with them. Rob, the VP Product Management, was a much better choice for talking to engineers than Steve, he didn’t rub them the wrong way and didn’t bullshit. Dennis, the CFO, was a voice of sanity. Marti, the VP Marketing, built a great 3-body marketing team that achieved amazing visibility with a very small budget, they were full of cool ideas and executed quickly and effectively.

A clear issue was the lack of domain knowledge in the executive team. Some of the false-start Sales VPs had it, the eventual Sales VP didn’t (but tried to made up for it by Steve-style bullshitting). Rob came from Wind River and as such the embedded systems domain, which is at least related. He had held a senior position at Wind River, but that didn’t prepare him for defining a product strategy for a startup. In the end he mostly managed the product development process, while the actual product strategy came from Benno.

But the most crucial shortcoming of the executive team was that all but Benno (and me, and maybe Dennis in his quiet way) were yes-men/women, who didn’t ever challenge Steve, and this presumably is why they were picked. No-one who is willing to form and express their own opinion, or dares to confront wishful thinking with facts, is welcome in the Steve world. Such a person would have to go (as it happened with a number of non-executive staff, see Tony and Josh).

Chicago head office was this bizarre perfect world, where everything went to plan and everything always looked good. Benno called it the “reality-distortion field” that was created there, with all those fantastic deals that were just about to happen.

Obviously, such an environment, where fantasy is reality and critical thinking is treason, is a recipe for disaster. It is also extremely painful for someone who spends considerable effort developing bullshit detectors in his students. Unfortunately I had to get used to muting my bullshit detectors whenever I went into a meeting with Steve. Meetings in Chicago were this roller-coaster of having fun catching up with folks, especially our marketing team, and facing this deluge of bullshit. Can’t imagine how others survived that on a daily basis.

The obvious question is, why didn’t I try to do something about this earlier on? It’s a question I still struggle with, I can’t find an easy answer. I can’t claim that I didn’t see the problems, but somehow I felt too insecure to react properly. I also didn’t know how to react, and I didn’t have good mentors to turn to. And I was isolated on the Board.


© 2014 by Gernot Heiser. All rights reserved.

Permission granted for verbatim reproduction, provided the reproduction is of the complete, unmodified text, is not made for commercial gain, and this copyright note is included in full. Fair-use abstracting permitted.

StartPreviousNext

OK Labs Story (5): Qualcomm

Our first customer was Qualcomm, and they were the most important one for a long time. During its lifetime, OK sourced the bulk of its revenues from Qualcomm. They were a critical partner.

The relationship was fantastic for a long time and, I’m told, quite unusual. Qualcomm doesn’t have a reputation of being an easy partner, particularly for small players. They also don’t have a track record of licensing technology, their usual approach is to by outright (technology or company) or do it themselves.

Throughout we maintained an excellent relationship at the engineering level. The technical people at Qualcomm held our engineers in high regard, I never heard anything other than compliments.

Steve’s original achievement was the re-negotiation of our Qualcomm contract, providing secure (and significantly increased) funding that was the stepping stone for creating the company. And he managed to increase the volume of the contract a few more times.

By that time, Qualcomm had long decided to base their future firmware versions on our L4, and, in fact, end-user products were close to shipping. Getting from there to deployment on a billion devices was now mere engineering, and we knew we wouldn’t fail on that.

However, over time, Steve’s style created problems with Qualcomm. He ended up seriously annoying a sequence of division heads (i.e. the people controlling the money), mostly through his negotiation style. As a company, we gained a reputation of arrogance, which really appalled everyone (who knew about it) in Sydney.

Negotiations, in true Steve style, always went to the brink, and all deals were last-minute. Essentially we were betting the company every year when the contract was due.

I vividly remember one particular Steve moment: The Qualcomm manager was hospitalised. As Steve reported at the Board meeting, he called him up “to wish him well” but at the same time remind him of our negotiations. Steve seemed mightily proud. But, as I soon heard through the engineering back-channel, the Qualcomm manager was hugely unimpressed.

This was on the back of an earlier disastrous misjudgment in dealing with Qualcomm. Google was moving into the mobile space with Android, and the first Android phone was the Nexus One, or HTC Dream. We knew it would have a Qualcomm chipset, with OKL4 the OS on the baseband processor. Steve thought it a smart idea to have a press release timed for the release, titled something like “Google had a Dream, and we’re in it”.

To say that Qualcomm was unimpressed about us preempting their PR that way is the understatement of the century. They were livid. They were foaming out of their mouth. Steve got a call saying “take that down immediately!” As Steve proudly reported at the board, they fumbled around with “communication issues” due to flights etc for a few hours to give it more exposure, until finally taking it down after about half a day. Fantastic way to treat your friends, and clearly a good way to make the relationship last, right?

And it was so annoyingly short-sighted and unnecessary. We could have simply waited for a few days, until Qualcomm had made their move. There was a lot of hype around Android, and it wasn’t going to die down in a day. We might even have got more attention by staying away from the peak of the hype, and in any case, by timing it less aggressively, Qualcomm wouldn’t have objected, the release would have stayed up, it would have come up in Google searches, etc. But long-term thinking isn’t Steve’s strength.

Needless to say, an increasing number of people inside Qualcomm had their knifes out on OK, and were waiting on the opportunity to stick them in. And the opportunities were bound to arrive. The main thing that kept the relationship going was the excellent work done by our engineers, which clearly provided value to Qualcomm. I talked more than once to senior people on the Qualcomm side who were torn between enjoying working with OK engineering, and pissed off with what they saw as arrogance and lack of trustworthiness of OK management. The net effect that influence inside Qualcomm shifted from our friends to our opponents.

Early in the life of the company we were introduced to the Qualcomm team which was working on their new DSP architecture, what later became known as the Hexagon. We met the chief architect and a number of other staff involved. Hexagon was to become Qualcomm’s proprietary replacement for ARM cores as their modem processors, and L4 was to be ported to that architecture.

On the engineering side the work made good progress. However, problems arose from multiple directions. One was Qualcomm-internal politics, where someone started developing their own (unprotected) RTOS, and started to sabotage us. But he was greatly helped by Steve’s waning popularity with Qualcomm management. After a long and painful period of politics and negotiations, Qualcomm finally decided to use the home-grown RTOS for the Hexagon.

Technically, this was a step backwards for Qualcomm, from a protected system to a traditional flat address-space model. But for us it meant that our days as the OS underneath Qualcomm’s modem firmware were numbered. Today (by all I know) L4 still ships on Qualcomm chips (so total L4 deployment should now be well over 3 billion), but on one or more of the auxiliary cores, no longer the main game.

While clearly not good, this wasn’t a disaster yet. Qualcomm had, throughout, received our code under the original open-source license (and could have just released it in source form). We didn’t earn royalties, only (lucrative) consulting revenue. This was one (but by far not only) reason for a complete re-design and -implementation of our kernel, which resulted in the OKL4 Microvisor, the basis of our virtualisation product. Essentially we now had two code bases, the “microkernel” for Qualcomm and the “Microvisor” for everyone else.

Our big chance to change the Qualcomm relationship arose when Qualcomm looked at supporting virtualisation on the application processor, and negotiated for a (non-exclusive) buy-out license of the Microvisor, to ship on their apps processors. Things seemed to go well, until Steve put an insanely huge asking prize on the table. Qualcomm was seriously pissed off, and the relationship never recovered. Internally our aim was to get just over 1/6th of the price asked. Most people involved believe that, had we asked what we really wanted, Qualcomm would have just accepted it without much negotiation. But with the price Steve asked, it was too bloody obvious that they could have done it themselves much cheaper. In the end we got nothing, except a terminally poisoned relationship.

The final straw came later, in the form of the AMSS disaster, which I’ll discuss in a future blog.


© 2014 by Gernot Heiser. All rights reserved.

Permission granted for verbatim reproduction, provided the reproduction is of the complete, unmodified text, is not made for commercial gain, and this copyright note is included in full. Fair-use abstracting permitted.

StartPreviousNext

OK Labs Story (4): The CTO

Benno’s appointment as VP Engineering, for all its good sides, was also part of Steve’s divide-and-rule approach – in this case designed to side-line me. It meant that Steve established a direct link with Benno bypassing me. I could also feel a distance developing between Benno and myself, which particularly saddened me, given that he was my star student and like a son to me. Whether this was the result of Benno thinking he had to actively compete with me, or it was Steve actively bad-mouthing me, I can’t tell (but I’ve seen a lot of bad-mouthing of third parties to consider the latter entirely possible).

Of course, there was also a (to a degree inevitable) clash between purity and engineering realities. And I felt that I had to give Benno the freedom to fulfil his role and earn the respect of the team, so I was very careful to avoid seeming at odds with him or undermining him. My confidence in Benno’s technical insight made this easier.

But one of the effects was that I was increasingly excluded from technical design discussions and decisions.

Circumstances made this relatively easy. Part of OK’s deal with NICTA was that the outcome of the still progressing research program (the seL4 microkernel and its formal verification) would go to OK, and I was to oversee the completion of that program. So I was 50% OK Labs CTO and 50% group leader in NICTA. As a part-timer I inherently couldn’t be running too much on a day-by-day basis.

In hindsight, this arrangement was a grave mistake. By the time seL4 was ready, OK had had to build its market and products on the existing technology, and wasn’t the right vehicle for marketing seL4. (That’s not the full story, but more about that later). We should have left seL4 out of the deal, and I should have either had a stronger day-to-day involvement or just be a consultant. But that’s with the benefit of hindsight…

So, my part-time status helped Steve to remove me from most decision-making. But I was excluded from too many other things too.

Many of the personal highlights of my OK experience were talking to customers, be it engineers, VPs or CTOs. I learned a lot about real-world trade-offs, market drivers and technical insights. And my presence helped the company’s reputation. One memorable example was a technical meeting at Nokia, with engineers and CTO-office staff, including a known virtualisation sceptic. As Abi told me afterwards, that sceptic after the meeting commented “he’s the smartest CTO I’ve met, and totally devoid of bullshit.”

But bullshit-freedom isn’t an asset in Steve World. In fact, he thought he could do anything I could do (just better, of course). For example he seemed to truly believe that having sat through one of my presentations, he could do it himself just as well. The few times I observed him doing this, I felt like hiding in a corner for embarrassment, and did my best to prevent damage (typically involving kicking him under the table). But it was surely embarrassing, and any engineers around couldn’t help noticing that. Steve simply can’t keep his mouth shut when he thinks he knows something, even if there are real experts around. I can only imagine what went on when he was on his own.

And I was around less and less. Once Steve hired a VP Sales (who sang to Steve’s song book without missing a beat) I was generally not taken to customer meetings any more. This was partially compensated by some of our top engineers moving into sales, such as Abi, another one of my star students. At least they knew what they were talking about, but generally weren’t in a position to take a stand against bullshit. The rare exceptions included field-application engineer Josh (also a former student) and sales guy Tony. The two of them brought in the biggest non-Qualcomm contract during my time at the company (from Motorola). In fact the two were the only successful sales team we ever had (despite Steve heaping praise on the VP Sales at the Board whenever they managed not to completely mess up a sale). But they didn’t take any bullshit, so they were pushed out and massively bad-mouthed. This became a pattern.

There were instances crying for my involvement, but mostly I only learned about them afterwards (or not at all). At one stage, several opportunities started to develop in China. One of our engineers, native Chinese, who was involved in the technical evaluations told me later “it’s because of you, you’re well-known in China” (something I was quite oblivious to, but had later independently confirmed). The natural reaction should have been to fly me over and talk about the technology and vision. But no, Steve and his VP-Sales clone knew better. In the end, none of the China opportunities went anywhere.

Another case was DoCoMo, the main Japanese mobile operator. They had, in a 2006 white paper co-authored by Intel proposed the concept of the dual-persona phone (two logical phones, work and private, on a single physical handset, separated by virtualisation). In 2009 they were looking at developing one. There was a competitor in play, whose only advantage over us was a security evaluation (similar to Common Criteria but extremely low-grade – glorified tyre-kicking). DoCoMo was going to go with us if we could tick that box. However, for us it would have been insane to go through the expense of an essentially worthless security “evaluation”. By the time I heard about it, they had been talking in circles for 6 months! I joined the next phone conference, and within an hour we had a solution. Too late, as it turned out, things had changed internally in DoCoMo and the project never happened.

There were plenty of other opportunities where I never got a chance to contribute. Besides Steve’s systematic side-lining of myself, a contributing factor was the cultural ignorance of Chicago (i.e. OK headquarters). Steve in particular prides himself of “understanding” other cultures, but, like is technical “insights”, this is very superficial stuff he’s read in mags and blogs. He utterly fails to understand the appeal of technical authority in continental European as well as Asian cultures. People there have high respect for professors and value their opinions, especially where the professor is strongly identified with a particular technology (as is the case with myself and the microkernel technology that formed the basis of OK products).

As such, the standard approach should have been to get me in front of senior engineers/CTOs of any prospective customer. Steve (and his VP Sales) never understood this, despite my attempts to tell them. But what do you expect from someone who seems to believe that bullshit is an adequate substitute for substance? One telling instance of complete lack of appreciation of culture was when he asked a junior marketing staff to approach leaders of big companies across a range of industry sectors. That way, the CEO of BMW got an email “Dear Norbert, I’d like to tell you about our virtualisation products”. The mail probably never reached Reithofer, but if it did, it could only get us filed in the category of dubious operators, rather than a respectable technology provider, and would certainly have done far more harm than good.

As a consequence I grew increasingly frustrated with my lack of ability to contribute in a meaningful way. This was compounded by my observation that the company strategy was wrong, based on wishful thinking rather than analysis of market needs and technical facts. (More on this later.) I was asked to write white papers and technical blogs (something I enjoyed greatly) but that’s a pretty minor role for the CTO and founder. And I was supposed to screen invention disclosures and work with the inventors and the patent attorneys on getting patens filed. That stopped after I assessed two of Steve’s “inventions”, which he got Josh to write up. After finding that the inventive step was non-existent, I wasn’t troubled with any further ones…

I also wasn’t effective as a director. I was caught between trying to avoid being seen as undermining the CEO while trying to voice my concerns. I failed miserably at the latter. Basically I was very naive (probably hard to believe by people who know me) but I was also quite aware of Steve’s bad-mouthing machine, which I had seen at work more than enough. I should have taken a stronger, independent stand at the Board, but that’s water under the bridge. It’s a reflection of my inexperience, which Steve exploited well.

More about the board later.

Basically, my position became more frustrating as time went on. Sometime during 2009 I had given up any hope that OK would be a major success. I stayed on partially because I thought I owed it to the many engineers who were there primarily because of me, and partially because, having been at the conception and birth, I wanted to stay around for the funeral. It couldn’t be far off.


© 2014 by Gernot Heiser. All rights reserved.

Permission granted for verbatim reproduction, provided the reproduction is of the complete, unmodified text, is not made for commercial gain, and this copyright note is included in full. Fair-use abstracting permitted.

StartPreviousNext

OK Labs Story (3): The Engineers

We had a world-class engineering team, by any definition. Except for one or two we hired from outside, the complete team that spun out of NICTA on 1 January 2007 consisted of my former students (and Carl, who never was a student at UNSW but had worked with us as an engineer since the early days of NICTA). Most had years of experience hacking microkernel code, and all had an excellent understanding of what makes a microkernel tick, and how you build systems on top of L4. Each could have doubled or tripled their salary by moving to Silicon Valley. They also had grown up working in a tight-knit team, so personnel-wise we were in good shape.

Before we could actually spin out, a (to me) unexpected hurdle arose: the contract negotiations. Steve took a very hard-line approach with the employment contracts. While most high-tech startups tend to provide fairly liberal conditions, what Steve offered was generally the legal minimum. In addition there were very broad and severe non-compete clauses (extending for a full year after leaving the company). While our legal advice was that this was almost certainly unenforceable in NSW, Steve insisted on them anyway.

The result was a staff revolt. About half of the staff refused to sign the contracts in the offered form, some making extensive changes. What followed was a lengthy (4 weeks or so) period of to-ing and fro-ing until we had a contract people were willing to sign. The end effect was that the company started on a sour note. I have to accept some of the blame here, as the original contract didn’t feel right to me, not the way I wanted to treat my students. But, again, I was inexperienced, didn’t trust my instincts in this unfamiliar territory, and trusted Steve to know what he was doing. Clearly a misjudgment on my side, and one I regret to this day.

In the end, Chuck, one of the A Team, decided not to join OK Labs, but instead moved to the Valley to join the Core OS Team at Apple, where he still works happily. At least he didn’t bear any grudge against me, we’re still good friends and meet up for a beer or three whenever I’ve got spare time in SF.

Not so for two others, Alex and Hal, who did sign on but never recovered from the original confrontation. They left about a year later, triggered by another poor handling of a situation by Steve: he had forgotten to perform a stock split before assigning stock options to staff, with the result that they got 2.5 times the percentage they were meant to get. Instead of saying “sorry, guys, we stuffed up” he tried to sell the correction as an improvement to staff, with a mail starting “good news! the price of your stock options has been reduced”. Not a smart way to treat grown-up engineers.

This created more bad blood. And, while I left these management issues in Steve’s hand (bad call, I know!) staff saw both of us as “management”. And it’s safe to say most staff were there primarily there because of me, and saw this as a betrayal of the trust they had in me. This was particularly the case with Alex and Hal, the two who left after a year. Hal hasn’t spoken a word to me since. It was my advanced OS course which had brought him to UNSW (as a postgraduate coursework student), so this hurts and saddens me.

My takeaway: If something seems wrong, do something, familiar territory or not. I should have known that, of course. One of the challenges was that I’m used to be able to trust my instincts, but this was an unfamiliar environment where I didn’t feel I could trust them, and Steve did a convincing show of knowing what he was doing. He also turned any doubt immediately into a personal trust issue, a great way to prevent any frank and open discussion. As I learned over the years, this was symptomatic: he would never admit to any mistake, and the mere suggestion that his approach wasn’t good would be seen as betrayal. The exact opposite to how I’m used to running things: frank and open discussions and looking for consensus.

After the bad start things improved quickly, and morale in the engineering team was very good once most people didn’t have to deal with Steve directly. Steve appointed Benno as VP Engineering, which helped to create that distance. Benno had been my star student, one of the most experienced in the team, despite his young age. He had proven himself in the early Qualcomm project, and is generally one of the sharpest people I know. Making him VP Engineering (rather than getting an experienced manager from outside) was a good move: Benno had more or less grown into the role already and did an excellent job on it. He is a real leader, respected by his peers, absorbed a lot about software processes, and also was a visionary Chief Architect.

We had excellent engineers, and a VP Engineering in charge who, despite his youth, provided strong leadership. The team achieved many amazing feats, generally delivering high-quality software very quickly. One of the most satisfying comment came from the Motorola engineer responsible for the virtualised phone project. He said “there were no bugs!” with the look on his face of someone who for the first time enters a Tardis and sees that it’s bigger inside. And we got consistently glowing feedback from customers after some of our engineers visited them, they never failed to impress through their insight and professionalism.

So in engineering we clearly had what it takes to be successful. All we needed was a plan and leadership to execute it, the engineers would build whatever was required.


© 2014 by Gernot Heiser. All rights reserved.

Permission granted for verbatim reproduction, provided the reproduction is of the complete, unmodified text, is not made for commercial gain, and this copyright note is included in full. Fair-use abstracting permitted.

StartPreviousNext

Follow

Get every new post delivered to your Inbox.