Re-routing the Router

By Jim Duffy
Network World, 09/04/00

When Layer 3 switches burst on the scene, industry pundits began predicting the demise of the slower, more costly router as a mainstay of enterprise networks.

Has that happened?

Not exactly.

True, Layer 3 switches are elbowing routers out of the core of enterprise nets for applications such as LAN segmentation and subnetting. In fact, the total market for modular and fixed Layer 3 switches has skyrocketed from a paltry $34.7 million in 1997 to more than $2 billion last year, according to Dell'Oro Group in Portola Valley, Calif.

Router commands

But WAN access and aggregation, multiprotocol support and voice/data integration are keeping -- and will continue to keep -- routers around for a long time in enterprise networks.

And the market numbers bear that out. Sales of high-end, data-only routers aren't exactly declining or even flattening out. In fact, sales were up 43% in 1999, from $1.6 billion the previous year to $2.3 billion, according to Dell'Oro.

The market for midrange voice and data routers, meanwhile, grew a healthy 60%, from $872 million in 1998 to $1.4 billion in 1999, Dell'Oro states. This is due to small and midsize companies looking to reduce monthly telephone charges by combining voice with data on the router network.

"What's often happened is that people that used to have a collapsed backbone routing configuration have replaced that with a Layer 3 switch," says Dave Passmore, research director at The Burton Group in Sterling, Va.

He adds, "But because Layer 3 switches have traditionally not supported WAN interfaces, they've repurposed [routers] and made them the WAN router. The router is no longer handling LAN-to-LAN traffic. That's being handled by the Layer 3 switch."

The Internet is also driving sales of high-end routers because service providers need more and more IP routers to accommodate traffic growth. Similar to what's happening with LANs, legacy routers are not going into the core of the 'Net, but around the edge, aggregating traffic from access points, tagging it with VPN identifiers and feeding it into a new class of gigabit'speed routers performing many routing functions in hardware.

Cisco rules the router market

When it comes to high-end legacy routers, Cisco is the dominant player, with 88.1% market share -- the only competitor of any consequence is Nortel, at 11.2% -- so Cisco's product road map pretty much determines how routers and switches will be used.

Investment giant Merrill Lynch is an example of a large corporation redeploying traditional Cisco 7X00-class routers as WAN aggregation devices. Last year, when Cisco introduced its Catalyst 6500 LAN routing switch -- which is what many would call a Layer 3 switch -- it changed the game for the 7X00 router, as far as Merrill Lynch was concerned.

Merrill Lynch's network serves 68,000 employees in 950 global branches. The network spans five continents, goes from 56K bit/sec dial-up and T-1 frame relay access connections to E-1 and DS-3 backbone links and consists of 1,800 to 2,000 Cisco routers, 45% of which are Cisco 7X00-class high-end devices.

"We've made a big investment in these 7500 series routers, but we don't think they-re a strategic investment -- that's why we-re not buying more of them," says Nicholas DeVito, director of technology infrastructure services at Merrill Lynch. "We-re pushing them more out to the WAN, and we-re replacing them as LAN routers with something like the 6500 series. We can deploy far more port densities in terms of high'speed Ethernet -- 100M bit/sec with Gigabit backbones" with the 6500s, he says.

If Merrill Lynch needs to buy more WAN routers for new sites, the company is inclined to go with Cisco 12000 Gigabit Switch Routers, which scale from 40G to 80G bit/sec and are targeted predominantly at service providers.

Even so, there are some pretty solid reasons why the 7X00 series routers will continue to have a place in the networks of Merrill Lynch and other enterprises.

For example, the 7500 is the only router or switch that supports Cisco Channel Interface Processor, a card that attaches to an IBM mainframe and connects legacy IBM SNA networks to IP networks without the need for an SNA front-end processor (FEP).

Merrill Lynch is looking to replace about 100 of those IBM FEPs with channel interface processor (CIP)-enabled 7500s. That may require the firm to continue purchasing 7500s, even if not for WAN routing, DeVito says.

"The 7500s are working well for that particular function, and given the data center mind'set, they like going with stuff that works," he says. "Over time, I'm sure we'll replace them, but in the short run, we-re happy with the way they-re performing. So we might buy a few more 7500s with CIP cards just for that very purpose. But by no means or stretch of the imagination do we see that as a major or significant investment; it's just pockets here and there where we need to supplement for performance or capacity."

Multiprotocol support

Another key feature of routers that keeps them around is the ability to route multiple protocols. Layer 3 switches usually support only IP, or IP and IPX, and do not include most of the software feature sets included in high-end routers.

'If you weren't running AppleTalk, you wouldn't need routers," says James Wiedel, director of networking at the University of Southern California (USC) at Los Angeles. "And if you weren't going anywhere, you wouldn't need a router at all with the Layer 3 switches."

Two years ago, USC replaced some old Cisco AGS+ routers with Layer 3 SmartSwitches from Cabletron. USC uses Cisco 7500s to route packets between networks and handle legacy, or non-IP, traffic. In all, USC has replaced seven AGS+ routers with SmartSwitches and two 7500s.

"The 7500s are doing WAN aggregation and routing random protocols, all the non-IP stuff," Wiedel says. "Our Layer 3 stuff cuts through the VLAN so that need not even touch the routers, unless you-re leaving a domain and going somewhere else. We-re using [routers] on the wide-area links because they do a little better job of not passing lots of traffic over the slow WAN links."

Wiedel says USC hasn't increased the port count of its 7500s "in years" even though the school's network has grown significantly. The network growth is being handled by Cabletron SmartSwitch 6000s with the company's SecureFast cut-through switching technology. In all, USC has about 30,000 switch ports, mostly 100M bit/sec and 1G bit/sec Ethernet.

"Right now, SecureFast has some real wins in it for us," Wiedel says. "It does a lot of the routing and it takes the load off of routers. That's not true of most other manufacturer's stuff that's running IEEE 802.1p and Q [frame't agging protocols]. They require a router to get in between the VLANs."

But even SecureFast cannot block out all of the broadcast traffic a flat, Layer 2 network is known for.

'There would be a lot of broadcast traffic that would carry over the WAN links that you really don't want out there," Wiedel says. "You still need the routers to shut up a lot of the noise."

Wiedel envisions keeping USC's two 7500s around for the short term. Longer term, though, they'll be replaced, he says. "At some point in time, they'll swap out for something else, but I don't know what that something else is."

Software giant PeopleSoft has an idea what that something is: Cisco's Catalyst 6500 routing switch with FlexWAN adapters.

FlexWAN debuted earlier this year as a way to "WAN-enable" the 6500 so it could serve as a single routing and switching platform to consolidate LAN, metropolitan-area network (MAN) and WAN services. By doing this, users can lower their cost of equipment ownership, simplify network design, ease network management, and migrate existing MAN and WAN networks, such as those based on routers.

To foster the migration, FlexWAN even uses the same T-1, T-3, ATM and packet-over'sONET WAN port adapters as the 7500 and Cisco 7200 routers. This seems like a simple, straightforward replacement for those traditional routers, which one would assume might now be ripe for retirement.

But Cisco stresses that the 7500 can still be used as a high-density enterprise WAN edge device with a "sweet spot" in T-1/T-3 aggregation. The box can also function as a virtual private network gateway and supports IBM SNA connectivity through the CIP. Cisco even added voice to the 7500 platform earlier

this year via a new Multiservice Exchange chassis that supports a circuit-switched bus.

But PeopleSoft's routers don't need to be heard; they need to be pumped up.

"We keep getting into oversubscription problems," says Stan Christensen, director of network engineering at the Pleasanton, Calif., software company. "We've got a lot of packet-over'sONET [interfaces] connecting a lot of the metropolitan-area networks together on those units as well. So we've had to keep adding multiples of 7500s just to keep them from oversubscribing too much."

PeopleSoft has been testing the 6500 with FlexWAN modules. With the 6500/FlexWAN combo, PeopleSoft can stuff as many packet-over-sONET modules as needed without oversubscribing the switch.

'We-re going to put it into testing and then production to see how it flies," Christensen says of the 6500/ FlexWAN combination. "If they do it right, it could easily replace the 7500. I don't see any reason why it couldn't. For us, it could pretty much replace everything we use the 7500 for."

Well, nearly everything. PeopleSoft is moving the 7500s out of the data center environment -- where big bandwidth lives -- and into remote sites for T-3 and E-3 aggregation.

"We've rolled more than half of our 7500s now into all of our major hub sites as we've been clearing them out of here," Christensen says. "They-re not useless; they-re just not good in a data center environment."

But the routing future, as PeopleSoft sees it, is clearly with the 6500/ FlexWAN combo.

"Since we've been going in these other directions of 6500s, we haven't needed to buy really much in the way of new 7500 product," Christensen says. "We-re still cruising on the products we've had for the last year or so."

WAN aggregation

So is Union Pacific Railroad in Omaha. Union Pacific is an old Wellfleet/Bay Networks shop with a router network composed of some 32 Backbone Concentrator Node (BCN) devices.

Union Pacific is not buying any more BCNs -- and has not for years -- because the company's long-term LAN routing strategy is to go to Layer 3 switching. But BCNs still handle a lot of the company's LAN segmentation chores because a migration project in a company the size of Union Pacific takes years.

And like other users, Union Pacific is pushing BCNs out toward the edge of its network for WAN aggregation duty.

"At the main locations, over half the users are still behind BCNs," says Brett Frankenberger, systems engineer at Union Pacific. "We were mostly token-ring. So it's not just a case where you can pull out the router and put in a core Layer 3 switch when you have to touch all of the end users. We have projects going on to do that, but there are probably 10,000 nodes at the main locations that would have to be touched. So it's not a one-year project."

Union Pacific is moving a lot of its servers off of BCNs to Fast Ethernet switches because "it's easier to touch them," Frankenberger says. Those servers are then linked to BCNs via Fast or Gigabit Ethernet.

The company's end users are still on 16M bit/sec token ring with an average of 50 to 100 users on a ring. The performance of these networks, interconnected through a BCN, has been "OK," Frankenberger reports.

Union Pacific's St. Louis operations are farthest along in the router-to-switch migration, Frankenberger says.

"There was a major PC replacement project in St. Louis, which is what drove that this year, because we-re going to be touching all of the desks anyway," he says. "We plan to redeploy the BCNs. The project in St. Louis is only going to get rid of half the BCNs there. There were four in the city in various buildings, and now there's still going to be two afterward to support the rest of the users. And there's a lot of WAN access hubbed out of St. Louis also. So we need that still."

About one-third of Union Pacific's BCNs are in Omaha and St. Louis. The rest of them are at major field locations. And right now there are no plans to migrate those field locations to Ethernet switching.

"They'll stay on the routers," Frankenberger says. "They generally have ATM links out of there, and the aggregate bandwidth leaving one of those locations is less than token ring anyway, so there's not a whole lot to improve."

Delivering voice/data routing

Dell'Oro predicts that more PBX and software-based voice functionality will be added to routers over time, which will trigger additional router sales.

As Domino's Pizza in Ann Arbor, Mich., looks to meld voice with its data infrastructure, its investment in Nortel's Access Node Hub (ANH) remote site routers could be capped as the company sizes up Nortel's Enterprise Edge product. The ANHs connect 32 remote sites to Domino's headquarters over 56K bit/sec and T-1 frame relay links.

Enterprise Edge combines digital telephony, IP routing and a LAN hub or switch all into one system.

"That looks like a pretty good solution for us," says Matt Maguire, Domino's director of IT. "It would be [a replacement for the ANHs] if we turned up a new office from scratch, but there's no reason to upgrade those unless their phone system is failing. We-re looking at it possibly for our stores as a low-end solution."

Enterprise Edge could replace both the ANH router and Nortel's Norstar key system at a remote site, Maguire says. Domino's is also evaluating a competitive offering from Vertical Networks. "They essentially have a Cisco router baked in it," Maguire says.

Down, but not out

So the role of the router is clearly changing.

Traditional, CPU-based, software-intensive routers, like the Cisco 7500 and the Wellfleet/Bay/Nortel BCN, may be a dying breed, but their function is still critical. Those functions will just be subsumed by a new generation of higher'speed, hardware-intensive routers that, for all intents and purposes, could be marketed as a switch.

"Traditional routers will, over time, be enhanced with their own ASICs and the like," says Eric Hindin, formerly an analyst at The Yankee Group in Boston. "I think they have plenty of life. If nothing else, there are so many of them out there, you and I will have long since moved on by the time there's no role for them in carrier and enterprise networks."

Related links

Contact Senior Editor Jim Duffy