Why AT&T's disaster recovery service failed

Basking Ridge, N.J. - If your frame relay service was knocked out by AT&T's network outage April 13 and 14, you might wish you'd relied on the carrier's formal Disaster Recovery Options (DRO) program for frame relay.

Don't bother kicking yourself about it. As it turns out, DRO didn't help.

The main reason: All three DRO options rely on the same frame relay network that failed to achieve backup.

"The thing about these services is, I'm sure [AT&T was] relying on other facilities going down, not the frame relay switches," said Steve Sazegari, president of Tele.Mac, a Foster City, Calif., consulting firm.

DRO comes in three flavors. The first, the Access Protection Option, lets users terminate two dedicated access lines into the same AT&T frame relay port. The service costs $250 to install and $2,500 to activate in case the first access line is cut (see graphic).

The problem: Because both access lines deliver traffic to the AT&T frame relay network, they would have been equally fruitless during the outage.

The second option, Back-Up PVCs, assigns two frame relay circuits of equal committed information rate (CIR) to the same route between two customer nodes. CIR indicates the amount of bandwidth the user has ordered for ordinary traffic rather than traffic bursts, which are subject to discard in case of network congestion.

The installation cost of Back-Up PVCs is nominal, but the activation fee rises to $5,000 in case of failure of the first permanent virtual circuit (PVC). The problem: A second PVC seeking a different route through the network would have failed because the whole network was blocked.

The last option, Growable PVCs, is the same as Back-Up PVCs except the two PVCs have different CIRs. The higher CIR of the second PVC is meant to deal with situations in which neither the carrier nor the user has forecast that an unusually high amount of traffic from one or more user sites will be offered to the network.

For Growable PVCs, the setup cost rises sharply according to the chosen CIR of the second PVC. For example, although it's $34 per month at the lowest level, 4K bit/sec, it increases to $725 per month for 384K bit/sec CIR. The problem: CIR size was not the issue during the AT&T outage because neither ordinary traffic nor data bursts could get through.

An AT&T spokesman confirmed that DRO did not enable users to get traffic through during the outage until the switches came back up anyway. He noted that AT&T offers dial ISDN backup but not as a specific frame relay option. Customers must order it separately, he said. Alternatively, users can buy similar services from regional Bell operating companies or competitive local exchange carriers.

"The lesson here is if you want a backup option, don't get it from the same carrier, or at least use a totally different service from that carrier," said Bryant Dunetz, president of CICAT Networks, Inc. in Fairfax, Va., one of Bell Atlantic Corp.'s largest independent ISDN installers.