Basking Ridge, N.J. - AT&T Corp this week identified the reason why its frame relay network crashed last week, blaming a series of unfortunate events for the network's 26-hour meltdown.
But users are on tenterhooks waiting for AT&T to prove a similar chain of events won't happen again.
Hanging in the balance are customer decisions about whether to install costly new backup circuits or move traffic to other AT&T networks or even other carriers. The telecom giant said the outage began when AT&T technicians updated software on a pair of T-3 interfaces in a Cisco Systems Inc. StrataCom BPX 8600 backbone frame relay switch. Frank Ianna, AT&T's executive vice president for network and computing services, said the upgrade triggered a preexisting software flaw -- which he did not detail - that caused the switch to start flooding the network with administrative messages.
In the coup de grace, the BPX failed to recognize the message storm and curtail it as the switch should have, and soon the network was swamped, blocking all customer data.
Ianna said the circuit card software upgrade, issued by Cisco, was simply meant to bring the software up to the current release.
And although the problem was with the software, Ianna added that AT&T's upgrade procedures were at fault. He declined to elaborate, but some observers questioned the decision to upgrade a component on a live switch during business hours.
"This was a calamity of errors," said Mike Schumacher, senior manager of enterprise network practice at KPMG Peat Marwick, in Houston. "The software patch should have been fully tested in a lab before throwing it out there." Other analysts were more concerned about the final leg of the three-part problem, in which the StrataCom switch failed to stop a storm of administrative alarms from coursing across the AT&T network of 145 switches.
"My biggest concern is that [AT&T] cannot predict the zillion different reasons that might create a message loop in the future," said Christine Heckart, vice president of telecom consultancy TeleChoice Inc. "That's a Cisco problem."
AT&T Chairman and CEO Michael Armstrong said he would continue to waive frame relay service charges for users until AT&T declared it was confident the problem would not recur. He expected that process to be completed by the end of this week.
But with some users continuing to see problems with the AT&T network, the carrier can expect pressure for even more credits from some users, especially those who had been counting on a recently announced AT&T frame relay service-level agreement (SLA).
For example, Bob Mason, chief information officer of computer reseller MicroAge Inc., in Phoenix, said last Tuesday - a full week after the outage - that eight of his 50 frame network nodes were out intermittently during the day.
"We have confidence in our supplier, but with another outage just yesterday, as opposed to a one-time event, it does give you pause for thought," Mason said.
In January, AT&T announced it would offer frame relay SLAs under which it would waive port and permanent virtual circuit fees for a month for a failed PVC that could not be restored in four hours or less. So far, the company has failed to tariff those SLAs.
Joe Lueckenhoff, AT&T vice president for data product management, last week conceded that AT&T will not start offering those SLAs until June, pushed back from the original March launch date.
Even MicroAge, which signed a five-year, $35 million contract with AT&T in 1996 for a variety of voice and data services, does not have a frame relay SLA in its contract. Other AT&T executives signaled that even when the SLAs arrive they may come with fine print. In a meeting with analysts, Rick Roscitt, president of AT&T Solutions, the carrier's outsourcing and managed-network arm, said his customers typically have custom SLAs under which AT&T gets credits for exceeding performance measures and debits for missing the measures. The two are netted out to see whether AT&T owes the user money.
Despite Armstrong's pledge, some industry veterans insisted that in the current situation AT&T will only end up offering credits for the actual period of the outage - essentially, a day's charges.
"The problem with [a simple credit] is that the rebate doesn't begin to touch the cost to our business," said Dan Rzonca, vice president of enterprise systems for MicroAge.
Roscitt did reveal that in the future AT&T will offer to pay customers' usage charges racked up on ISDN services because of an AT&T-caused network outage.
Ianna calculated that AT&T's stated frame relay network availability parameter of 99.99 percent would allow it only 50 minutes of downtime per year. "From our measures, we kind of shot that this year," he said.
