Clemson's computational colossus

High-performance 35-teraflop grid built on blade servers, InfiniBand storage

Peer inside one of the country's premiere university computational research centers and what will you find inside?

Peer inside one of the country's premiere university computational research centers and what will you find?

Clemson data center secrets

Balance.

That's one key message James Leylek, executive director of the Clemson University Computational Center for Mobility Systems (CU-CCMS), wants to deliver about the unusual place he oversees. The CU-CCMS provides a massive computational infrastructure for private sector companies in the automotive, aviation and energy industries.

On a more operational level, a second key message is no matter how much compute capacity in this data center, CU-CCMS doesn't stand a chance of meeting its mission without the InfiniBand network underlying it.

Research at the speed of industry

Leylek's brainchild is designed to balance industry's need for fast, timely results delivered on budget against the academic requirements of the master's and doctorate candidates conducting the research. The private sector and academia had long been butting heads over the need for speed vs. the academic mindset of slow, deliberate research leading to the publishing of scholarly papers, Leylek says.

Leylek, who has worked on both sides of the fence, wanted to find a way to put these tensions to rest. He envisioned a research center run under the auspices of Clemson, but staffed by career engineers rather than students. The engineers would develop unique mathematical models that would go above and beyond what any company could get from commercial software packages for attacking big physics challenges.

Three years ago, Leylek's vision edged toward reality. With matching dollars from Clemson's home state of South Carolina, which sought investments in high-tech, knowledge-based activities, the university committed to the idea. The engineering team immediately began designing a "balanced" data center infrastructure, Leylek says.

James Leylek

As part of its balancing act, the CU-CCMS computational infrastructure would need to be capable of dealing with extremes. "We knew we would have a mix of software that could only make use of a single CPU for simulations to nearly linearly scaling software that could run on up to 10,000 CPUs and everything in between," Leylek says.

"Imagine hundreds of millions of billions of equations solved hundreds of thousands of times -- that's what we needed to do," Leylek says.

Teraflops galore

CU-CCMS engineers work their math magic via a 35-teraflop computational grid composed of 43 Sun Blade 6000 Modular Systems, Sun Fire servers and Sun StorageTek systems. The grid comprises 3,440 processing cores, hundreds of gigabytes of RAM and 14 terabytes of storage, Leylek says.

While 35 teraflops (1 trillion floating point operations per second) may not seem like much oomph in the supercomputer community, CU-CCMS engineers actually have far more crunching power than peers elsewhere, Leylek says. At CU-CCMS, each engineer has access to a dedicated 8.75 teraflops "around the clock, no strings attached." Compare this with the University of Texas at Austin's Texas Advanced Computer Center, which divvies up 579.4 teraflops among 1,500 users. That's roughly 0.4 teraflop per user, he explains.

On the CU-CCMS grid, the servers interconnect with each other, the blade chassis and the storage over a 20Gbps InfiniBand network built using Voltaire's Grid Director switches. On top of that, CU-CCMS has superimposed a 120-TB parallel I/O storage system from Panasas. "Otherwise, we knew we'd be I/O bound one day," Leylek says.

The InfiniBand network is really the heart of it all, says Leylek, noting that the Voltaire switches have been infallible. "This is a beautiful thing because without the InfiniBand network working extremely reliably we'd be in deep, deep trouble. With all the computational grid and now parallel I/O, we're really sticking our necks out doing things in timeframes that shock even me," he says. "Of course we need experienced people, and the software and computer power and so on, but the network works the hardest of all components here."

Realizing the dream

And this is early days for the Clemson Computational Center. Today the grid and InfiniBand network sit in a pre-existing data center that Clemson was using for other purposes before Leylek snagged the space. "The data center had 21,000 square feet of unused raised floor, a good bit of which was covered with carpet and office space," Leylek recounts.This whole project started with revamping that data center, bringing in new uninterruptible power supplies, diesel generators, air conditioners and physical security systems to get the existing infrastructure up to snuff. The CU-CCMS opened with seven teraflops of compute power and a 10Gbps network, both of which quickly grew to their current states without growing the data center footprint.

Leylek says he's got plenty of expansion space in the data center, even as he eyes rapid growth that will take CU-CCMS to 100 teraflops soon. The university has put plans in place to build the CU-CCMS a new, 100,000-square-foot data center, which Leylek says he hopes will be online within two years.

"Slowly but surely we're realizing our dream. In the middle of this global downturn and recession, we're expanding and I can't contain my excitement about what will happen when the economy turns up," Leylek says. "We're not going to expand but explode."

Schultz is an IT writer in Chicago. She can be reached at bschultz5824@gmail.com.

Learn more about this topic

The do-it-yourself supercomputer

Top 500 supercomputers: Welcome to the petaflop generation

IBM supercomputer to heat university buildings

Insider Tip: 12 easy ways to tune your Wi-Fi network
Join the discussion
Be the first to comment on this article. Our Commenting Policies