Lustre high performance file system to support InfiniBand

What do systems from Bull, Cray, Dell, HP, IBM, Linux Networx, SGI and Sun all have in common? The Lustre file system.

Lustre is the Linux-based open source file system developed by Cluster File Systems (CFS), a small Seattle firm. It is a parallelized file system used in commercial high performance applications (such as life sciences, digital video and aerospace) and in numerous supercomputing environments (including the Department of Defense, the National Science Foundation, NASA and the Swiss National Supercomputing Centre). 

While some vendors rebrand the file system (HP calls its SFS, for example), it is essentially identical across all the devices that use it. Lustre is a 64-bit file system that provides for high bandwidth data transfers and the next release is expected to support clusters of more than 5,000 nodes and 400 I/O servers. The largest Lustre cluster in the field today supports 4,000-nodes, although 6,000-node installations are reportedly under development.

As a 64-bit file system, Lustre offers several advantages over 32-bit environments. Most obviously, it can handle the largest binary large objects (BLOBs) – file sizes and I/O requests, for example, would be limited only by the available address space. The total number of files within the system would have a theoretical limit of 134 million files per directory, with an unlimited number of directories, both of which are clearly well beyond the practical limits for such things.

Lustre most often runs on TCP/IP over Ethernet, but with the new release, the company has added support for InfiniBand. CFS claims performance of 115M byte/sec using Gigabit Ethernet, 550M byte/sec with 10 Gigabit Ethernet, and 600M byte/sec using InfiniBand.

One particularly interesting thing about Lustre is the way it is distributed. CFS has always followed a two-phase release, with the first distribution shipped to customers and then, some months later, a second (but identical) distribution being made available to the free software community. With the next release, scheduled for December, this policy changes in one significant way: future versions will be released to customers and to the general public simultaneously.  While it is hard to know exactly what the impact is likely to be on the open source community, it certainly seems likely that they would benefit from getting their hands on source code sooner than was previously the case.CFS Web site. 

For engineering, scientific and other applications requiring very high performance, 64-bit operating systems and file systems are a real advantage. If this is of interest, you might visit the

*** Wish list for Santa ***

Remember to send in your suggestions for this year’s storage wish list for Santa. In the best of circumstances, the annual wish list can be a useful way for you to be an activist and let the vendors know what you need. In less positive conditions, it can be a good way to blow off steam.