Donald Becker started the Beowulf Parallel Workstation Project in 1993 at NASA Goddard Space Flight Center. The project's goal was to cheaply mimic the computing power of expensive mainframes and supercomputers with clusters of commodity hardware and free operating systems. The effort at NASA was named after the eighth-century Danish poem Beowulf, who slew mighty beasts - in the case at NASA, those beasts were supercomputers and mainframes.

This evolved into the Beowulf Project, which became popular among researchers for linking together Linux and FreeBSD machines to take on huge computational tasks. Linux-based clusters are now frequently seen in the list of the top 5 supercomputers in the world. Becker founded clustering software firm Scyld Computing, which was acquired by Penguin Computer. He is now Penguin Computer's CTO. I recently spoke with Becker about how far clustering has come.

Q: How have Linux clusters evolved?

A: The original Beowulf clusters had a full [operating system] install on each machine. It was as difficult to run as a set of workstations. If you were running a slightly older version of software on your desktop, or had different versions of software across machines, it was a disaster. Often you ended up with computation results that failed with no useful error messages.

The size of the clusters has also increased. Clusters of 15 to 20 machines were the norm when we started out, now you typically have clusters of 50 to 100 machines, up to over 1,000.

Q: How has cluster management improved?

A: Now we're moving to scalable computing - being able to dynamically scale what you're doing ...  We're leveraging the software [Scyld] introduced four years ago and refining it ... The key idea is that instead of a fixed-sized cluster ... You now have [a] machine that has [a] master, where you can control multiple servers from a single point.

Q: Will clusters totally replace supercomputers?

A: We're not aiming to replace high-end supercomputers. There are still applications that require those types of machines. But for 80% to 90% of the [computer-intensive] applications out there, you can do [the] same work on [clusters of] PC class machines for far less money.

Q: What are the things clusters can't do?