Take this with a grain of salt.
Northern Telecom, Inc., the giant Canadian manufacturer of switching gear, says it has reduced recurring network costs 25% by replacing its T-1 wide-area infrastructure with an integrated voice/data/video T-3 ATM backbone based on - surprise, surprise - its own switches.
We expected a prettied-up version of the story, butany time a $13 billion company says it has found a way to shave $18 to $25 million per year off network expenses - even as its network traffic quadrupled - we have to pay attention.
While Bill Holtz, vice president and general manager of Global Enterprise Services and David Hilger, director of Enterprise Network Services at times made it sound like migrating the huge network to ATM was as easy as falling off a log, they did ultimately share some hard earned lessons.
When the idea to migrate to ATM came up in 1994, the company wanted Holtz to approach the project as if he were a customer. He had to build a solid business case and sell it to the CEO, chief operating officer and the president.
The parameters: The project had to have a payback of two years or less. The transition had to be transparent to users. The case could not be based on "green dollars," meaning expected productivity gains couldn't be used to offset costs. And Holtz and his team had to use street prices for the Nortel gear.
It took three months to build the case, and when it was finally approved, six months to implement the network and only 14 months to reap the payback, Holtz says.
Part of the savings came from revamping the IT organization. Where the company once had 1,400 people scattered among separate WAN, LAN and other teams - each with separate planning, equipment sparing and other expenses - all of those groups were collapsed into a single Enterprise services group of 900 people.
But the bulk came from savings on T-1 multiplexers, the T-1 circuits Nortel was leasing from Sprint Corp. and MCI Communications Corp., and from savings on long-distance phone calls between corporate sites.
The international network Holtz was looking to overhaul comprised about 100 T-1s supporting traffic generated by 63,000 employees at 220 locations.
Roughly 20% of those sites accounted for 80% of traffic, so these high-density locations were the sites that were brought onto the ATM backbone first.
The Nortel backbone is anchored by eight large Vector ATM switches at strategic regional locations which are, in turn, linked to 81 Nortel Passport edge switches. The Passports support DS-1 interfaces to voice switches and a range of data interfaces, including Ethernet, FDDI and V.35.
Nortel used DS-3 links between major sites and the ATM cloud of service provider WilTel, which was bought out by WorldCom, Inc. during the course of a three-year contract. All access lines are on redundantly routed Synchronous Optical Network (SONET) rings to ensure network availability.
Nortel pays a flat fee regardless of how much traffic gets sent, a pricing option Holtz is high on because he knows exactly how much he is going to spend each month. He also considers the ATM price a bargain. "We bought DS-3 ATM circuits from WilTel for less than we were spending on T-1 links from a range of carriers," Holtz said.
Into the fire
So what did they learn?For one thing, you need a well-thought-out plan and prepared people. Holtz says it is important to send a core team of IS staffers through rigorous ATM training. In Holtz' case, that meant recruiting switch designers as educators, an advantage few others could expect.
That team was then charged with getting input from every major corporate group that would be affected by the migration. "Program management will make you or break you," he said.
From this effort popped an unforeseen problem: a language barrier. It took time to make sure everyone understood each other. For example, quality of service has a specific meaning in the ATM world, but it can be interpreted generically by the uninitiated. "The worst problem was not knowing the jargon of the questioner," Holtz said.
Another momentous problem, and an ongoing one at that, is waking applications up to the fact that they are connected to an ATM network.
To ease the migration to the new infrastructure, the dedicated links that SNA and other legacy applications used were simply emulated in the ATM world by nailing up connections. But that defeats the ability of ATM to dynamically allocate bandwidth and reduces network efficiency, so Nortel has been wrestling with the arduous task of tweaking those applications.
The job is larger and more costly than anyone had expected, Holtz admits. In fact, the cost of updating applications was not factored into the original business case, when, in retrospect, maybe they should have been.
Application problems are as basic as this: Where programs used to rely on the net to handle flow control, with ATM the network passes the problem back to the applications. "It's a different world," Holtz says.
Every application has to be evaluated in terms of how they tolerate network delays, what turnaround times they need for responses, and how they behave when links go down.
"In many cases, people flat don't know the answers to these questions," Holtz says. Some traditional TDM modeling tools were used to discover what they could, and many new modeling tools were developed on the fly. In the end, however, the network designers often simply had to specify ranges.
Routed IP
Nortel also resorted to the nail-up trick when it moved its giant intranet into the ATM realm. Since then, however, the company has implemented variable bit rate (VBR) service on links between some routers, allowing those devices to burst up to 10M bit/sec.That change resulted in the unexpected: "When we installed VBR, we uncovered a bunch of pent up demand," Holtz says.
That demand hasn't been too taxing because so far only 20% of the routers that had T-1 WAN links are allowed to burst. Before he can let the rest burst, he has to better predict how that will affect total network loads and figure out if the network can handle it. In the meantime, the rest of the routers are supported on T-1 permanent virtual circuits (PVC).
As things stand, if router spikes do begin to cause network congestion, voice calls spill over onto the public phone network to release bandwidth, but that hasn't happened yet.
The network, in fact, is overbuilt to accommodate all this nailed-up capacity. As more applications and routers are given the green light to grab bandwidth only when they need it, the network will become more efficient. Translated, that means Nortel's existing infrastructure will be able to accommodate future traffic growth.
Today, the ATM network is the first path of choice for voice, data and video from roughly 70% of the company's North American sites.
In terms of their satisfaction with carrier ATM services, the only major carrier glitch they experienced was also a major embarrassment for Nortel: WilTel had to turn off the ATM network once to swap out its Nortel switches for Cisco Systems Inc.'s Stratacom BPX.
Longer view
Holtz' goal now is to extend the ATM net globally. While he doesn't anticipate problems in rolling it out to Nortel sites in the Pacific Rim and Europe, South America will be a problem because of the poor infrastructure there. And that will have an impact on what carrier the company signs up when its carrier contract expires this year. "I need a global supplier because my business is global," he said.He already is looking at cranking up backbone bandwidth. When the contract is rebid, he will try to get 155M bit/sec OC-3 links for the same price he is paying now for 45M bit/sec DS-3s.
So can you take Nortel at its word?
Even if only half of the savings are real, they are nonetheless remarkable. The unknown is whether you would squander those savings trying to make ATM work. After all, Nortel had rooms full of ATM engineers ready to help see this massive project through.
