WorldCom outage raises new doubts

WorldCom is not the first to see its Internet backbone buckle after suffering a network outage, but customers and analysts are concerned the carrier’s nine-hour slowdown is an indication that more troubles lie ahead for the financially beleaguered company.

CLINTON, MISS. – WorldCom is not the first to see its Internet backbone buckle after suffering a network outage, but customers and analysts are concerned the carrier’s nine-hour slowdown is an indication that more troubles lie ahead for the financially beleaguered company.

WorldCom’s UUNET backbone slowed to a crawl Oct. 3 after a technician made a configuration change to operating system software that runs on some of the carrier’s border routers. While WorldCom confirmed these facts, the carrier would not provide more details about the nature of the outage or how it fixed the problem.

One customer remains frustrated by the dearth of information coming from WorldCom, which filed for bankruptcy in July and has laid off 16,000 workers since.

“We got in contact with WorldCom early Thursday morning after noticing the network was sluggish. They disclosed they had a router or switch issue, but we’ve never been debriefed,” says Greg August, CIO at the Cystic Fibrosis Foundation in Bethesda, Md., which runs a 50-site IP VPN over WorldCom’s network. “We don’t need them to waste time during the crisis, but some reports afterward would be helpful.”

While August waits for WorldCom to explain the outage and what the carrier is doing to assure it does not happen again, he says he’s concerned bigger problems are looming.

“The scary thing is that they could be losing control of some sites and that’s why they haven’t explained what happened,” August says. “I doubt it, but I really don’t know why they wouldn’t come clean.”

Latency across WorldCom’s network was up around 900 msec and packet loss exceeded 22% during the carrier’s outage, according to Matrix Network Systems, a company that monitors Internet performance for enterprise users. The Internet typically operates with about 50 msec of latency and about 0.05% packet loss, says Tom Ohlsson, vice president of marketing and business development at Matrix.

The packet loss and latency that WorldCom experienced during its outage wreaked havoc for some WorldCom customers. The World, a Boston ISP, lost all connectivity to the Internet for about three hours primarily because the ISP’s main connection is through WorldCom.

“Other customers that were multihomed to five or six ISPs did not see the same types of outages, but only slowdowns,” says Bill Palumbo, CEO at Matrix.

The faulty software that WorldCom uploaded to its border routers caused some routes between cities to completely disappear. This problem spread throughout WorldCom’s UUNET network in the U.S.

“This is not something we predicted because we thought WorldCom would have [been guarding] against a very public, major network catastrophe,” says Michael Suby, an analyst at Stratecast Partners. “Questions are now raised in the minds of customers if this outage was related to the reduction of people or the company’s bankruptcy. It’s hard not to draw those conclusions.”

At the very least, the outage exacerbates public perception that WorldCom now has more on its plate than it can handle.

“It just adds to [WorldCom’s] cycle of problems,” Suby says.

Border router outages that affect an ISP’s route tables could happen to any carrier, but nine-hour outages are not typical.

“It did take quite a bit of time to start seeing improvements,” Palumbo says. “In other instances when we detect a route has been removed and we [notify the carrier], it takes 15 to 30 minutes to fix the problem.”

“Whatever they did [originally] caused more erratic and geographically dispersed outages,” he says.