by Jonathan Corbet

Kernel space: The enterprise real-time feud

News
Dec 11, 20075 mins

Red Hat and Novell developers work together on improving the kernel’s real-time performance. But outside the idyllic calm of the linux-kernel mailing list, marketing bluster continues.

Jonathan Corbet At the end of November, LWN posted a pointer to Novell’s announcement for its SUSE Linux Enterprise Realtime offering. The resulting comments were surprisingly negative. Some readers took exception to the language of the release – though it really is just the standard tortured English which is seemingly required for press releases. But others question the need for realtime response in “enterprise” settings. Anybody who is still wondering about the value of that product will be doubly confused now that Red Hat has announced a realtime distribution service of its own. Clearly somebody sees a need for deterministic response in big corporate deployments.

What’s going on here is that corporate operations are, increasingly, being run by automated systems. One immediate example is in the financial trading field, where automated systems execute customer trades and, often, make the decisions to perform the trades in the first place. Often the conditions that make a particular trade advantageous last for very short periods of time – perhaps only as long as it takes for the first interested party to arrive on the scene. So predictably fast response to trading decisions is an absolute requirement. Losing too many milliseconds in the execution of an order can cost real money.

It does not take much imagination to see that, as these systems become more capable, more corporate dealings will happen via automatic agents which require lightning-fast response. So enterprise realtime has the look of a growth industry. It’s not surprising that the two companies most interested in selling Linux-related services into the enterprise market have announced offerings within a week of each other.

What is surprising is the amount of silly sniping which has come with these releases. Consider this quote from the Red Hat side:

“Till last week, Novell sold a Real Time system that forked their kernel,” said Scott Crenshaw, VP of Red Hat’s infrastructure business unit, on a visit to London today. Last week, Novell announced SUSE Linux Enterprise Real Time (SLERT), using open source Linux code that Crenshaw says was “80 percent” written by Red Hat people. “We welcome Novell to the real-time community,” he said, slightly stingingly: “We look forward to them making contributions to it.”

Or from the Novell side:

Note to Red Hat: this is open source, remember? Novell is shipping tested and enterprise-hardened Linux with real time capabilities. Just because Red Hat is again late to market (see enterprise Linux desktop, Xen virtualization, etc.) doesn’t mean Linux contains “beta code.”

Despite their competitive relationship, Linux distributors have traditionally dealt with each other in a friendly, even cooperative manner. At the development level, things are still that way: developers for a given project work together and only very rarely does anybody care about who a given developer’s employer is. Developers, it seems, are more polite than managers and PR people.

So who is forking the kernel? In fact, both distributors will be shipping something which is pretty far from the mainline. Back in October, LWN looked at the contents of the realtime tree, finding some 400 patches which have not yet made it into the mainline. Anybody who is shipping a true realtime kernel will have to include the bulk of those patches, and probably some others as well. In recent years, much work has been done to enable distributors to ship kernels which are much closer to the mainline, but these realtime offerings are a step in the opposite direction. They are, for all practical purposes, forked kernels.

That statement should not be taken as a criticism; there is no other way to ship realtime Linux at this point. While much of the realtime code has been merged, some of the deepest, most necessary components remain outside of the mainline. The process of getting those patches merged has taken quite a bit longer than anybody would have expected; among other things, some of the core realtime developers have been distracted by small side projects like the i386/x86_64 architecture merger. Until the process of getting the realtime patches into the mainline runs its course – something which could happen over the next year – anybody shipping realtime distributions will necessarily have to roll their own kernels.

More than almost any other area of kernel development, the realtime code has been the subject of recurring debates over who deserves the credit for the work. See this LWN article from 2005 for an example. This time around, Red Hat would appear to be claiming ownership of the realtime work. In fact, much of this work, including the crucial low-level preemption work which got the current realtime effort going, was done at Red Hat. But other components have come from companies like MontaVista, Linutronix, TimeSys, and, yes, Novell (and others, of course). For these two companies to be arguing about credit is a little silly; both are clearly significant contributors to the kernel (and beyond).

We may see more of this kind of talk, though. This market looks like it could be big, so the companies working in that area are going to make a serious effort to be successful there. The result may well be that Linux ends up as the dominant system in the fast-moving, agent-driven world where much of corporate operations appears to be heading. That cause will be helped, though, if the relevant managers and spokespeople take a clue from the developers who are making all of this work actually happen. We are all building this system together; pointless mud slinging can only get in the way.