AT&T tests disaster response

* AT&T completes its latest disaster recovery exercise in Washington, D.C.

With hurricane season facing U.S. businesses, AT&T completed its latest in a series of regular tests of the carrier’s ability to respond quickly to natural disasters and major outages.

AT&T ran a network disaster recovery exercise outside Washington D.C. on June 7 and 8. AT&T conducts these exercises four times per year in different cities around the country.

In these exercises, AT&T simulates large-scale disasters and network service disruptions, and then the carrier’s employees work to restore communications to customers as quickly as possible.

The Washington, D.C. area exercise featured 24 trailers, which house network equipment that mimics the technology in a central office switch. Forty AT&T employees participated in the exercise.

"We can duplicate any of our central offices in the trailers," says Ken Smith, director of operations for AT&T’s network disaster recovery team. "We keep off-site copies of all software configurations as well as the engineering piece in centralized repositories."

AT&T officials wouldn’t say how much the Washington D.C. exercise cost. However, AT&T says it has invested $300 million in its network disaster recovery program during the last 10 years. The program includes specially trained managers, engineers and technicians from across the United States as well as a fleet of more than 150 equipment trailers and support vehicles that can support voice and data communications.

AT&T says its network disaster recovery team has been activated 21 times since 1990, including responding to the September 11, 2001 terrorist attacks in New York as well as last year’s Hurricane Katrina in Louisiana and Mississippi.

AT&T officials said they have refined their disaster response efforts after each incident. For example, the September 11 terrorist attacks were the first time that AT&T’s network disaster recovery team was unable to fly into the region involved in the disaster.

"Until 9/11, we made the assumption that whatever disaster we faced we could fly people in from all over the U.S.," Smith says. ``Another thing that’s changed for us since 9/11 is that we learned an environment with a destroyed central office will not necessarily be a clean office. It could have anthrax in there. So now we have a fully functional hazmat team. We fill our own air tanks. We don’t clean up the office, but we can go into a condition and do telephone-level work."

With Hurricane Katrina, AT&T officials learned that they needed to have their own housing available rather than renting hotel rooms. AT&T bought three campers that its employees could use for showers and sleeping while responding to Katrina.

"Until Katrina, we always made the assumption that housing - hotels and food - would be available," Smith says. "When we got down there, no housing was available. Our employees slept on the floor of one of our command centers for four nights until we bought the campers."

With such extensive disaster recovery capabilities, AT&T is confident that it can respond to most outages.

"With Katrina, we lost four offices. Even though four offices were under water, our software capabilities were able to restore within an hour 80% of the traffic that was knocked out because of flooding. The other 20% we had to do physical restoration," Smith says, adding that most network traffic was restored in 24 hours.

For more on AT&T’s disaster recovery capabilities, click here.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Now read: Getting grounded in IoT