Data center networks often exclude Ethernet

It’s not often that Ethernet is on the outside of an emerging network technology market. But in linking some data center equipment to high-speed pipes, Ethernet sometimes has its nose pressed to the glass door, looking in.

For data center network managers, server interconnect technology falls into two distinct camps. For most, Ethernet, the world standard for networked computers, is how Windows, Linux, Unix and mainframe boxes are plugged in and accessed. But in the rarefied realm of high-performance data center clustering, technologies such as InfiniBand and some niche, proprietary interconnect technologies, such as Myricom’s Myrinet, still have a strong hold.

Over the past several years, InfiniBand switches have emerged as an alternative for some users. Makers such as Voltaire and Infinicon came to market with high-speed clustering switches that connect servers with specialized host bus adapters (HBA). These systems can provide as much as 30Gbps of throughput, with latency as low as the sub-200-nanosec range. (By comparison, latency in standard Ethernet gear is sometimes measured in microseconds -- one millionth of a second -- rather than nanoseconds, which are one-billionth of a second). This server-to-switch technology was so attractive that Cisco purchased InfiniBand switch start-up TopSpin a little more than a year ago for $250 million.

To continue reading, register here and become an Insider. You'll get free access to premium content from CIO, Computerworld, CSO, InfoWorld, and Network World. See more Insider content or sign in.

It’s not often that Ethernet is on the outside of an emerging network technology market. But in linking some data center equipment to high-speed pipes, Ethernet sometimes has its nose pressed to the glass door, looking in.

For data center network managers, server interconnect technology falls into two distinct camps. For most, Ethernet, the world standard for networked computers, is how Windows, Linux, Unix and mainframe boxes are plugged in and accessed. But in the rarefied realm of high-performance data center clustering, technologies such as InfiniBand and some niche, proprietary interconnect technologies, such as Myricom’s Myrinet, still have a strong hold.

Over the past several years, InfiniBand switches have emerged as an alternative for some users. Makers such as Voltaire and Infinicon came to market with high-speed clustering switches that connect servers with specialized host bus adapters (HBA). These systems can provide as much as 30Gbps of throughput, with latency as low as the sub-200-nanosec range. (By comparison, latency in standard Ethernet gear is sometimes measured in microseconds -- one millionth of a second -- rather than nanoseconds, which are one-billionth of a second). This server-to-switch technology was so attractive that Cisco purchased InfiniBand switch start-up TopSpin a little more than a year ago for $250 million.

A need for speed, and more

“Ethernet is a good, versatile technology that can handle almost anything,” says Patrick Guay, vice president of marketing for Voltaire. “But Ethernet never had to address the levels of [traffic] efficiency and latency” required in clustered computer systems, storage networking and high-speed server interconnects, he adds.

“It’s not that there is no place for 10G Ethernet in data centers,” Guay says. “There is just a certain subset of customers who need more than what Ethernet and IP offer.”

This was the case at Mississippi State University’s Engineering Research Center (ERC), which runs several large Linux clusters used in engineering simulations for defense, medical and automotive industry research, among other areas. The ERC’s Maverick is a 384-processor Linux cluster connected by Voltaire InfiniBand products. Voltaire’s Intros 96-port InfiniBand switch is used to connect the diskless processor nodes, which access storage — and even operating system boot images — over the InfiniBand links.

This lets Roger Smith, network manager at the ERC, set up cluster configurations on the fly; however, many processors needed for a task can be called up quickly.

Smith says the communication between nodes in the cluster is "very chatty with very small messages" being passed back and forth. This requires extremely low latency among the nodes, so that no messages are missed, which could disrupt a job running on the cluster.

“Ethernet was just not ready for prime time, to get to the low-latency needs in some data centers” over the last few years, says Steve Garrison, marketing director for Force10 Networks, which makes high-speed Ethernet data center switches.