How did Linux come to dominate supercomputing?

After years of pushing toward total domination, Linux finally did it. It is running on all 500 of the TOP500 supercomputers in the world, and who knows how many more after that. That’s even more impressive than Intel’s domination of the list, with 92 percent of the processors in the top 500.

So, how did Linux get here? How did this upstart operating system created by a college student from Finland 26 years ago steamroll Unix, a creation of Bell Labs and supported by giants like IBM and Sun Microsystems and HP, Microsoft’s Windows, and other Unix derivatives?

“There wouldn’t be Linux if it weren’t for Unix,” said Steve Conway, research vice president for Hyperion Research, the high-performance computing (HPC) unit of IDC. “The Unix era gave way to the Linux era because Linux is more open and not vendor-specific. So, here was a chance with Linux for the whole community to have one main flavor of an operating system.”

None of the major Unix flavors supported the x86 architecture, either. Sun did with SunOS, which was a text-based OS, and it had Solaris on x86 but never made a big push for it. All the other Unixes were on custom RISC processors. Of course, no one saw the massive rise of x86 on the server, either.

Prior to Linux, the only heavily supported x86 Unixes out there was The Santa Cruz Operation with Xenix and FreeBSD out of the University of California at Berkeley. But Xenix was a desktop OS, never a server OS. By the time it sold out to Caldera Systems in 2001, its opportunity had long since passed and Linux was already gaining momentum.

Then there was Microsoft. It came out with clustering software as early as Windows NT 4.0 but made its first real effort in 2006 with Window Compute Cluster Server 2003. However, it never made much of an effort in that area, and by this decade it had folded some of the clustering technology into standard Server edition.

“For their part, Microsoft took aim at HPC for a couple of years but didn’t put a lot of wood behind that arrow, as it were,” said Conway. “They weren’t alone. It didn’t seem at the time that the HPC market was going to be as big as it became. This was the pre-cluster time, for the most part. Up to the 90s, the HPC market was worth $2 billion, with everything thrown in. Last year it was $22 billion.”

NASA helps HPC take off

What made HPC take off? NASA, whose job is to explore space, made an amazing Earth-bound advancement. In the mid-1990s a team of programmers came up with a way to cluster x86-based servers for collective processing power for a cheap alternative to expensive, proprietary HPC systems at the time. Dubbed Beowulf, it was not tied to one particular operating system, any free and open source OS would do. But the inventors used Linux, and that started the momentum.

“What really carried Linux more than anything was the arrival of the cluster in around the year 2000,” said Conway. “That’s when clusters really entered the HPC market and the appeal was commodity technology, including Linux. Through the decade, the HPC market grew at a compound rate of 20 percent.”

Beowulf supported FreeBSD, so why didn’t FreeBSD take off? Conway attributes it to being one of those technologies that was a very good idea but just didn’t catch hold in large part because they weren’t promoted very well.

Linux has something FreeBSD doesn’t have: Linus Torvalds. Torvalds is a tough, demanding leader. Many say too tough. He has a nasty streak that can put Steve Jobs to shame. But he has been the leader Linux needed.

Vendor support for Linux

The final piece of the pie was vendor support, something FreeBSD never had. Linux had organized companies behind it. I remember attending a computer fair in Hartford, Connecticut, in 1993 (where all the local screwdriver shops would set up tables and sell components for system builders) and seeing Bob Young hawking CD-ROMs of the very early versions of Red Hat Linux. Boy, if I had known then what I know now…

Anyway, Red Hat took off and helped drive Linux in a way UC Berkeley never did for FreeBSD. Eventually came Caldera, SuSe and Canonical. Then came the big dog: In 1999, IBM announced support for Linux. At that point, Unix was a dead OS walking. It just didn’t know it.

Linux, and Linus, didn’t get here alone. It stands on the shoulders of AT&T/Bell Labs, Dennis Ritchie, Ken Thompson, Keith Bostic, Richard Stallman (yes, I must give credit where it is due, however annoying he can be), Sun, HP, IBM, SGI, and many more characters and vendors. But it really does stand tall.