One big subnet

10G Ethernet holds endless possibilities for seamlessly extending enterprise networks into the MAN and across the WAN.

What can an enterprise do with the equivalent of more than 6,000 T-1s worth of bandwidth? Just about anything.

Once 10G Ethernet becomes the backbone of choice for service providers, it opens the door to powerful possibilities. 10G Ethernet provides a natural handoff between corporate and carrier networks without the expense of encapsulation or conversion to SONET, frame relay or ATM. Terabytes could zip across networks like instant messages rather than the imperceptibly moving glaciers they often resemble. LAN, metropolitan-area network (MAN) and WAN connectivity would become one big subnet.

With 10G in the LAN, MAN and WAN, a car maker handling designs for multiple new models could handle backups and database synchronization among Detroit, Munich and Tokyo sites without downtime or waiting for one location’s off-peak evening hours. Trillions of bits could traverse the globe easily, either as native Ethernet, packet over SONET or packet over wavelength division multiplex (WDM). Virtual LANs could come and go as fast as project phases were completed.

Suddenly, SANs would become as large as enterprise networks, server and database access would no longer be a function of geography, and surplus capacity could be shifted and exploited as needed. If one location suffers a natural disaster, all of its applications and data could be replicated at a different site within minutes.

End-to-end quality of service (QoS) would mean that executives’ e-mail could be given priority over end-of-quarter sales numbers hogging most of the available bandwidth, regardless of user or server location. A LAN becomes a WAN, and a WAN becomes the world.

But this lofty vision will have to wait, as full 10G products are only now becoming available and services remain non-existent. The technology is pricey, and some technical shortcomings must be addressed before it scales to WAN thresholds.

Today, most 10G Ethernet ports come in blades for 1G switches. Because of this, throughput typically tops out at about 8G bit/sec. Carriers and equipment vendors point to 2004 as the real starting date for 10G Ethernet. That’s when leading vendors such as Cisco, Extreme Networks, Foundry Networks and Nortel expect to have switches that perform at 10G bits/sec. And next year will be when Gartner has forecasted that 10G volumes will begin to triple, year after year, going from 5,000 ports sold in 2004 to 185,000 in 2007 (see graphic “Exponential Ethernet”).

That’s the same time frame in which carriers such as AT&T, Verizon and SBC will have 10G services ready. “I have a placeholder in my plan for next year to be able to fund some of the activity and tests,” says Franco Collochio, Ethernet services product director for AT&T. “Late 2004 is our development time frame.”

But early adopters, ranging from bandwidth-starved corporations to competitive local exchange carriers, aren’t waiting.

“T-3 [45M bit/sec] was not enough bandwidth for us,” says Hal Marietta, director of network services for Liberty Healthcare Group, in Port St. Lucie, Fla. “We decided to do more than 1G since we knew we wanted to do SAN replication at some point, without impacting the standard data flow.”

Compliance with the new Health Insurance Portability and Accountability Act regulations, in their stringency regarding security and availability, is driving bandwidth demands, he says. To handle the 20 terabytes of data that Liberty Healthcare needs to be backed up every week, Marietta has been using Extreme’s BlackDiamond switches at three sites. He has trunked together multiple 1G ports for the backup.

The 10G mesh

Ethernet’s established familiarity as the lingua franca of networks is propelling it into MANs and WANs. Ethernet has evolved from 10M to 100M to 1G and now 10G bit/sec in a little more than 25 years. The 802.3ae standard for 10G Ethernet, approved about a year ago, has a range of 25 miles, a far cry from the shorter hops required by its predecessors prone to signal fade. With a longer reach and a throughput that matches the OC-192 threshold of SONET and WDM backbones, 10G Ethernet becomes a means for corporate and carrier networks to mesh more seamlessly than ever.

That in turn creates the possibility of virtual networks – known variously as transparent LANs, VPNs or Ethernet subnets – on a scale previously unworkable. It leverages the 250 million Ethernet ports already in service and enables turbo applications such as grid computing using terabytes and petabytes of data, or SAN replication, with which backup and disaster recovery can occur in seconds. And QoS, largely absent from any switched Ethernet equation in which prioritization schemes differ among users and carriers, finally could be offered across WANs.

“A metro-area service would only give us the ability to have a site located farther away, but WAN-level could definitely help us,” says Liberty’s Marietta. “We’d have no bandwidth limitations, so it wouldn’t matter where the server was. We can place resources where it makes the most sense.”

That’s the same line of reasoning used by network engineers at the Information Sciences Institute (ISI) at the University of Southern California in Los Angeles. “We look at 10G as an enabler,” says Richard Nelson, director of computing at ISI, which has been using 10G Ethernet for more than a year.

Initially, ISI deployed the 10G to handle an increase in high-performance desktops and computing clusters. But USC decided that kind of speed made it possible to mirror the university’s computing resources at ISI. USC has installed tape drives and other storage gear at the ISI location, 18 miles away from the main campus, so that if service or access to the campus is disrupted, it can bring the network back up from the data stored at ISI.

Using Foundry BigIron Gigabit Ethernet switches, ISI and the university are linked by a private 10G MAN that also connects to an Internet peering point in Southern California. From that location they tap into the TeraGrid network, a five-site, 40G grid-computing backbone that connects supercomputing and research centers across the country. “Whether it’s an earthquake shake table or electron microscope data, we need the capacity for tera-scale research,” Nelson says.

Nelson says he can foresee data center migrations and shipping tapes around for backup and restoration becoming passé. “Instead, you do a bulk transfer over a 10G network. People with terabytes of storage don’t want to do it at 1G – they want 10G or faster,” he says, adding that he’s looking forward to actual 10G switches to avoid some of the complexity of trunking multiple 1G connections, especially because the lack of trunking standards preclude multivendor 10G networks.

Vendors needed time to perfect 10/100 Ethernet, and the same holds true for 10G, Nelson says. “As interface costs come down and availability of the right kind of fiber increases, 10G will become more normal in the corporate world. People will start to see the possibilities with remote backup or as a less-expensive alternative to [SONET] OC-192,” he adds.

Paying the price

10G potential is rich, but so is its price tag. Industry analysts place the price per port for 10G blades in the $30,000 to $40,000 range, but prices have gone as low as $10,000, according to Brian Strachman, senior analyst at In-Stat/MDR. In the first quarter of 2003, vendors shipped a total of 170 ports of 10G equipment. Contrast that with 734,000 ports of 1G Ethernet with an average cost of $518, and it’s clear why 10G has yet to catch fire.

Adoption of a 10G Ethernet-over-cable standard, expected for completion by early 2004, should drive prices down and increase uptake of 10G, Strachman says. Gartner expects average port prices will fall from $19,500 in 2004 to about $7,710 by 2007.

Pricing for 10G services is a little trickier. Consider that Cybercon.com offers 1G services at $3,950 per month. This does not include local loop charges, which can range from $600 to $7,500 or more, depending on bandwidth size and distance from the carrier point of presence. Carriers won’t likely charge 10 times as much for 10G services, but enterprise users can expect to pay at least double the prices of 1G Ethernet. AT&T, Verizon and SBC all say it is too early to discuss pricing. 

10G has some other drawbacks. Critics like to point out that 10G Ethernet is not as resilient as SONET, and most companies are not going to tolerate a best-effort WAN service. Most carriers are looking at Multi-protocol Label Switching or the Martini specification for handling packet over SONET to make Ethernet behave more like SONET. How carriers manage those techniques as they scale up on the WAN remains to be seen.

As with any other service, enterprise users should be sure to negotiate a service-level agreement that establishes minimum-performance thresholds and compensation for any glitches.

The QoS mechanisms for 10G could stand improvement, says Mike Knoll, central office group leader for Hancock Telecom, which has been using 10G Ethernet blades from Extreme for about a year. His company offers cable television from one switch, and Internet access and voice over IP on the other switch. It’s using 2G bit/sec of its own fiber, lighting up more strands as needed.

As it stands now, users set up QoS by designating percentages of a fiber or connection that an application, user or location can use. “We need tighter bandwidth instead of percentages,” says Knoll, adding that the ability to designate 10% of a 10G pipe doesn’t really offer much additional value. “We need to be able to pick a [set amount of bandwidth].”

A billion bits per second from the campus to the wide area will open up lots of big possibilities, but it’s going to take at least a year for 10G to grow into the fullness of Ethernet’s next logical exponent. Apart from time and money, there’s no reason why the internetworking of corporate and carrier networks across the WAN should falter. Completion of a few more technical specifications should help 10G’s adoption, which would help push prices down, while the first few services start to become available.

10G will widen the WAN like never before.

Sweeney is a writer and editor in Los Angeles who has covered IT and networks for 20 years. He can be reached at terry@tsweeney.com.

The Hulk, courtesy of ILM ©2003 Universal Studios. All rights reserved.