The Lustre HPC file system has another new home — and it’s the right one

After years of bouncing from one company to another, the Lustre file system that is so popular in high-performance computing (HPC) has been sold yet again. This time it went to an enterprise storage vendor. Finally, it’s in the hands of a company that makes sense.

DataDirect Networks (DDN) announced it purchased the Lustre File System from Intel this week as Intel looks to pare down non-essential products. DDN got all assets and the Lustre development team, who are undoubtedly relieved. The announcement was made at the International Supercomputing Conference (ISC) in Frankfurt, Germany.

Lustre’s history

Lustre (which is a portmanteau of Linux and cluster) is a parallel distributed file system that supports multiple computer clusters with thousands of nodes. It started out as an academic research project and was later acquired by Sun Microsystems, which was in turn acquired by Oracle.

Oracle more or less abandoned Lustre, and it fragmented among several organizations. One of them was Whamcloud, which was acquired by Intel in 2012. Intel did make a decent effort to sell and maintain Lustre by setting up a new business unit and a new product, called Intel Enterprise Edition for Lustre.

However, last year Intel said it would cut back in Lustre development, so selling it was the best thing for it because HPC users feared that without support from Intel, Lustre would fade or be forked.

“Intel did a really good job of selling it off in a way that Luster will be well taken care of,” said Steve Conway, COO and senior research vice president at Hyperion Research, a division of IDC that specializes in HPC.

DDN’s plans for Lustre

Now it’s in the hands of DDN, which specializes in storage for HPC and exascale systems and has used Lustre throughout its product lines.

“Intel was a good steward, but it’s been musical chairs as far as owners go, kind who wants to take care of it,” said Conway. “DDN’s a good home. It needed a good home, since Intel decided it didn’t make sense to keep it.”

While Lustre is the largest player in parallel file systems used in HPC, it has also migrated over to the enterprise world into financial services, healthcare, energy and manufacturing, as well as others that use clusters.

“When clusters moved to the enterprise, Luster moved with them,” said Conway.

But DDN doesn’t have Lustre all to itself. This version of Lustre is hardly the only version of the OS. There are other flavors from vendors such as Cray that are based on the Intel version, so they aren’t too different. And there are consortiums that have some say in the development of the file system, such as the Open Scalable File Systems (OpenSFS) and European Open File System (EOFS) groups.

DDN may be a storage company, but they have a very holistic view of things. It has added embedded big data software, distributed cloud and object technology, support for NVMe, and GPU acceleration to its systems.

“Over the next few years, we are going to invest significant resources to enhance usability and to broaden Lustre’s capabilities and feature set in the direction of flash-enabled performance, analytics, enterprise, and hybrid cloud,” said Alex Bouzari, founder and CEO of DDN, in a statement.

DDN will operate the Lustre team as an autonomous division within the company, ensuring full and sustained support of the Lustre customer and partner ecosystem. The division will be run by Robert Triendl, senior vice president of global sales at DDN, who played a major part in DDN’s involvement with Lustre development over the past decade.