AT&T frame relay net goes down for the count

Carrier may be liable for huge SLA penalties.

By David Rohde and Sandra Gittlen
Network World Fusion, 4/14/98

AT&T's frame relay network has fallen and it can't seem to get back up.

As of 1 p.m. today, AT&T said it had restored service to 96% of its frame relay customers.

The outage began at 3 p.m. Monday and continued through the evening. The situation affected not only business-to-business data communications but also a wide swath of the consumer economy. Bank ATM machines, travel agency orders and credit card transactions were disrupted.

Many customers reportedly switched to backup systems running over private lines, ISDN or other carriers' networks, alleviating some of the disruption.

Nevertheless, the outage may trigger big penalties in AT&T's frame relay service-level agreements (SLA). Frequently negotiated with individual customers, AT&T's frame relay SLAs went mainstream in January as the carrier loaded them into its tariffs.

Under the new standard SLAs, AT&T guarantees to provide 99.99% availability of its frame relay network from service interface to service interface. The carrier also guarantees to restore lost permanent virtual circuits within four hours, or the customer gets the affected ports and PVCs free for a month.

At a press conference today, AT&T Chairman and CEO C. Michael Armstrong said the company, which relies on frame relay for about $1 billion each year, would not charge customers for frame relay service until the problem has been isolated, confirmed and a fix has been identified.

Armstrong said the cause of the outage has not been identified, but it has been narrowed down to two out of 145 nodes that were exchanging information. The AT&T recording pointed to possible problems in a Los Angeles switch. AT&T's frame relay network runs over StrataCom BPX switches from Cisco Systems, Inc., and the spokeswoman said "we are working closely with Cisco" to resolve the outage.

Also, AT&T Executive Vice President Frank Ianna said that the network management software that comes with the switches did not alert AT&T of the problem in enough time to ward it off. "We did not have the time we would have liked to react to the situation," Ianna said during the conference.

A problem in one switch can affect much of the network because PVCs are virtual circuits whose paths may vary depending on route utilization. The spokeswoman could not specify when total service would be restored.

A letter was on its way this morning to AT&T's 6,000 frame relay customers from Armstrong. "I want to apologize for the outages you are experiencing in our frame relay service," Armstrong said in a statement. "Network reliability is one of our highest priorities and this situation is clearly unacceptable to you and to us."

The apology may not assuage everyone, however.

"I lost $4 million of revenue today due to [the] frame communication outage," said one user in a posting on Network World Fusion. "I understand that all AT&T InterSpan frame customers and the entire AT&T internal administration were out of service. The outage was so devastating that I now have to reorder 50% of my carrier service and diversify into Sprint and MCI."

The user speculated that because "AT&T has segmented their net into three areas it might be architecture limitations which were violated during normal switch port installation."

Another user criticized AT&T's response to the problem, calling it "pathetic."

"This national outage by AT&T affected every single customer who relies on this service, causing countless problems behind the scenes at major corporations as well as affecting consumers who were shopping," the user said.