Frame fiasco puts heat on MCI WorldCom

By David Rohde
Network World, 08/16/99

You could call it the great MCI WorldCom Frame Relay Brownout of '99, but in a sense you'd be wrong.

Insiders say the congestion problems that plagued an estimated 3,400 of the carrier's 14,000 frame relay customers last week really should be called the Legacy WorldCom Network Brownout of '99. While the old WorldCom frame relay net bogged down, MCI's frame network actually stayed up through the crisis.

But the fact that the merged company was unable to switch traffic from the WorldCom network to the MCI net indicates how far MCI WorldCom still has to go to integrate its global data platforms.

And although the congestion problem stemming from a botched Lucent switch upgrade was largely fixed by the end of last week, users blasted MCI WorldCom for not responding quickly and failing to communicate the nature of the problem.

"If WorldCom could have routed traffic to the MCI switches, that would have helped," says one Midwest user whose 30-node legacy WorldCom network went down while his 10-node MCI network stayed up. "But in our opinion they couldn't do it even if they wanted to."

Analysts wondered how far MCI WorldCom could now push on pending network projects, such as its plan to unite around an Ascend Communications (now Lucent) switch architecture.

"This raises the question of the integration of the different networks," says Rosemary Cochran, vice president of Vertical Systems Group, a consultancy in Dedham, Mass. "If you have legacy MCI customers who are happy, you don't want to touch them. But then do you have features that are available to one subset of customers and not the other?"

MCI WorldCom's congestion problems began Thursday evening, Aug. 5, as technicians loaded new software into Lucent switches that anchor the old WorldCom frame relay network. Although MCI WorldCom won't specify exactly how, the upgrade appeared to interfere with congestion-control mechanisms and caused excessive latency on customers' permanent virtual circuits (PVC) - the links between carrier-network switches.

Many customers with higher-speed PVCs then had so many frames dropped that the event began to look like an outage. In about half the affected customers' cases, some of the Data Link Connection Identifiers - frame relay PVC network addresses - were destroyed altogether and had to be rebuilt during the course of the week, according to reliable sources.

Before the recent problems began, customers were told to expect an overnight "Emergency Cascade 9000 software upgrade" throughout the legacy WorldCom network, both domestic and international. "The duration of the outage should be no longer than one minute per circuit," said the text of the alert, which was obtained by Network World. Cascade 9000 is an old name for a frame relay switch now owned by Lucent following its recent purchase of Ascend.

An MCI WorldCom spokeswoman says that after significant congestion on the night of Aug. 5, users were notified of the problem on Friday and again on Monday. But several users say it was the other way around, with users opening trouble tickets overnight in response to alarms and then encountering a wall of nonresponse.

For example, San Francisco-based insurance company State Compensation Insurance Fund saw anywhere between four and 12 of its 28 WorldCom frame relay sites go down at a time for several days, according to WAN manager Rod Taitano. Despite using an automated maintenance system from an outside vendor that opened and updated trouble tickets with WorldCom starting Friday morning, "I finally got my first fax Tuesday from my account rep," Taitano says. The representative marked the memo's subject line "Oops!" and went on to describe in general terms what Taitano already knew: The network wasn't working right.

Adds Washington, D.C., attorney Hank Levine, who negotiates user contracts with carriers: "MCI WorldCom is being amazingly unresponsive. The customers are going completely and totally ape."

Yet most legacy MCI users - whose traffic travels over Nortel's BayStream routers in a network dubbed HyperStream frame relay - reported no problems. For example, one of the Chicago Board of Trade's automated systems went down because it uses a WorldCom-affiliated network, but other financial users - such as Nasdaq - were unaffected.

This duality caused network managers nationwide to scramble to explain to their corporate executives what was going on, after the general press began reporting that MCI WorldCom's "global frame relay network" was on the fritz.

For example, Allen Lund Co., a transportation broker in La Canada, Calif., suffered nary a blip in its 16-node HyperStream network. "It's been a great network for us," says Ken Lund, the company's network manager.