What's beyond 10G Ethernet?

40/100G Ethernet seen as 2012 phenomena at earliest; is 400G coming before Terabit Ethernet?

So 10G Ethernet is now taking off - 10G accounts for roughly 25% of the overall $18 billion Ethernet switching market and the growth trajectory is only up from there.

Also read: Ethernet everywhere!

Which begs the question: What's next?

Even though 10G has hardly reached maturity or saturation, its uptake will undoubtedly usher in the next wave of Ethernet switching. And that will be 40G/100G and eventually Terabit Ethernet, though there may be intermediate steps in between.

Indeed, the 40/100G Ethernet standard was recently ratified and products are now emerging on the market. Extreme Networks, Force10 Networks and BLADE Network Technologies have all announced switches support 40G Ethernet ports for data center, applications - such as aggregating all those growing 10G ports.

And Cisco, Juniper, Brocade and Alcatel-Lucent have all announced or begun shipping 100G Ethernet interfaces on their service provider routers. Verizon and the Internet2 research network are using Juniper's while AT&T is evaluating Cisco's.

On the chip side, Broadcom just unveiled devices for 100G Ethernet interfaces but designed to scale up to Terabit Ethernet across the backplane. Fulcrum just announce a 40G Ethernet chip design at the Supercomputing show Nov. 16 that it claims supports more than 1 billion packets per second of performance - or enough for up to 72 ports of 10G Ethernet at full line rate.

Bar chart showing growth of 10G Ethernet

And Intel plans to put 10GBase-T directly onto standalone and rack-mount server motherboards next year, bringing even lower cost 10G connectivity directly to the server. This will significantly ramp 10G adoption and increase demand for 40/100G.

"Everybody is talking about (40/100G Ethernet) as a data center future and they want to know what our plans are," says Bill Seifert, CTO of Avaya's Data Solutions business. "In 2011, most CIOs are going to be picking their vendor for 10G switching in the data center. They're going to need some aggregation trunks to feed from the top-of-rack to the core switches."

Aggregation of 10G is one requirement for 40/100G Ethernet. Others are virtualization and network convergence.

Data centers are implementing server virtualization to increase utilization and resource efficiency, share workloads and decrease power consumption. Server virtualization is putting more application load on fewer servers due to the ability to decouple applications and operating systems from physical hardware. And, as virtualization and the movement of virtual machines around the infrastructure accelerates, so too must the network.

More application load on less server hardware requires a higher-performance network.

In concert with this is a migration to a unified data center switching fabric that consolidates LAN data with storage traffic. These unified switching fabrics converge storage protocols onto Ethernet, which requires a high-speed, very low latency, lossless architecture - one that's traditionally been found in data center SAN networks.

In service provider networks, demand for 100G Ethernet is being fueled by video. According to Juniper, two-thirds of consumer traffic on networks will soon include some form of video. And Cisco says that 90% of Internet traffic will be video by 2014. Using 100G, HD movies can be downloaded in seconds and terabytes of scientific data can be simultaneously shared by laboratories in two different hemispheres.

But these are not overnight sensations. Vendors say it will be 2012 or 2013 before 40G finds sustained uptake in data centers.

"The driver for 2011 is a continuation of server virtualization that's driving 10G interfaces at the edge, which will drive 40G connectivity at the core layer," says Stephane Robineau, vice president of product management at Alcatel-Lucent. "In 2011, probably some will start [on 40G]. But I'm expecting [growth] in 2012."

To get a jump start, Alcatel-Lucent may unveil a 40G Ethernet module for its OmniSwitch 10000 in 2011, according to Robineau.

A competitor agrees on the timeline for 40G adoption.

"I don't think [40G] will be mainstream until 2012," says Saar Gillai, vice president for advanced technology and CTO of HP Networking. "We're seeing massive growth in the cloud of 10G at the edge. The price has to be reasonable and the density has to be useful. Until you have that, you have things that are gimmicky."

Extreme Networks says it is offering four-port 40G modules for its Summit X650 switch at $1,000 per port, but that does not include optics. With optics, prices for 40G Ethernet on single mode and multimode fiber approach $8,000 per port, industry players have said.

For 100G, those prices can be as high as $100,000 to $800,000, vendors have said.

Gillai says HP will have offerings in the 40/100G Ethernet area in the next 12 months.

After 40G will be 100G, even though some users could use 100G now. The New York Stock Exchange is looking to reduce latency and increase bandwidth as much as possible to expedite delivery and dissemination of market trading data. NYSE has just begun testing 100G Ethernet for its data center network.

"We're going to be very early adopters if we roll this out," says Andy Bach, senior vice president and global head of communications for NYSE Euronext, the owner of the New York Stock Exchange. "We'll use 100G to go between switches in the data center and then (we may) use 40G to go to the servers. That'll be next on our docket.

"If you look historically, we have grown bandwidth roughly on the order of 20% per year, compounded," Bach says. "You just follow that line out and it continues to grow, and you get to terabit fairly soon."

But NYSE has yet to discuss 100G Ethernet prices with its vendors, Juniper and Force 10. Pricing of 40/100G Ethernet will play a key role in data center demand, almost as much as the need for the raw bandwidth.

"This part of networking - Moore's Law completely applies," says Dhritiman Dasgupta, product marketing director in Juniper's Fabric and Switching Technologies Business Group. "We wanted to time this in such a way that it's an economically feasible product that we can sell."

Juniper is looking to sell 40G Ethernet at a "small premium" over what the company charges for 10G Ethernet ports, Dasgupta says, without disclosing when Juniper will have 40G and 100G Ethernet on its switches.

Cisco too would not discuss when it will ship or how it will price 40/100G Ethernet on its Nexus 7000 data center switches. Cisco did demonstrate four-port 40G Ethernet line cards on its Catalyst 6500 switch at both the Ethernet Alliance and Supercomputing conferences. (Also watch: Inside Cisco's data center.)

But the transition from 10G to 40G "will take some time" because 40G will come in at first generation prices, says Martin Hull, product marketing manager, Cisco Data Center Solutions.

Cisco, Juniper, Force 10 and all major switching players claim their high-end core chassis switches are 40/100G Ethernet capable, with per slot capacities over 100G. Force 10's CTO John D'Ambrosia chaired the IEEE 802.3ba task force, which defined the 40/100G Ethernet standard, and says the building blocks, such as 25Gbps electrical signaling, are in place to extend the current 802.3ba standard to other media and then to higher speeds. D'Ambrosia started a new study group last month to investigate 100G Ethernet backplane and copper cable interface applications that could lower the cost of 100G and then provide a schematic for boosting speeds beyond 100G.

"There are a lot of initiatives underway right now to come out with technologies related to making a lower cost, higher density 100G that will then be leveraged to the next speed," D'Ambrosia says. "There will be a huge debate at the next speed. The question is, how are we going to do it? Plain and simple."

The 25G signaling and 100G copper backplane work might make 400G Ethernet the next logical step, D'Ambrosia says, citing conversations he's had with system and component vendors. Using 25G signaling over 40 fiber pairs to get to terabit is just too unwieldy, he says.

"There's no study group for Terabit Ethernet," D'Ambrosia says. "And there are no projects underway right now to do 50G electrical signaling."

But cost is still the big driver in growth of high-speed Ethernet. People will not jump to 100G if it doesn't make economic sense, D'Ambrosia notes.

"It's only going to be higher capacity at the right cost," he says. "You can't look at those issues separately. It's nice to say all of this stuff but at the end of the day everybody wants to make money."

Copyright © 2010 IDG Communications, Inc.

The 10 most powerful companies in enterprise networking 2022