WorldCom's network struggles

WorldCom's Internet backbone was buckling under the strain of a "network issue" most of the day, which caused severe packet loss and delays for users in the U.S. and overseas.

Latency across WorldCom's network was up around 900 milliseconds throughout the day and packet loss exceeded 22%, according to Matrix Network Systems, a company that monitors Internet performance for enterprise users.

The Internet typically operates with about 50 msec of latency and about 0.05% packet loss, says Tom Ohlsson, vice president of marketing and business development at Matrix.

WorldCom's network outage started at about 7:00 a.m. ET after technicians uploaded new software to the carrier's border routers, Ohlsson says. WorldCom would not confirm these details for Network World, but Ohlsson says his group was given detailed information about the cause of the outage directly from the carrier.

A WorldCom's spokeswoman did say that the carrier is in the process of restoring all of its customers and that technicians and WorldCom vendors are investigating the cause of the outage.

Latency and packet loss started to improve around 4:00 p.m. Matrix's Ohlsson says this nine-hour network slowdown is "unprecedented in the brief but significant history of the Internet."

"A number of carriers are going through tough financial times, but WorldCom has laid off thousands of employees. The average workload of a WorldCom engineer is four to five times what it was a year ago," he says.

WorldCom let up to 16,000 employees go after it http://www.nwfusion.com/news/2002/0722wcom1.html filed for bankruptcy protection in July.

Even if other ISPs would have needed nine hours to restore service to most of their customers after a similar outage, WorldCom will carry an additional burden because most will tie its outage to its financial situation and its smaller staff.

Customers throughout the U.S. and in Europe experienced high latency through most of the day, Ohlsson says.

Ohlsson says WorldCom technicians uploaded faulty software to thousands of border routers on WorldCom's network. But the technicians did not know the software had a glitch until the routers were simultaneously upgraded at 7:00 a.m.