Could you or your outsourcer handle a one-two disaster punch?

* Recovering from back-to-back disasters

I'm sure you have read quite a lot about disaster recovery and business continuity following Hurricane Katrina. I originally had no plans to add to that body of work. However, as I am writing this, Hurricane Rita is a day or two from the Texas coastline. Back-to-back disasters can pose some interesting problems for business continuity. Having participated in the recovery of a major data center operation because of a disaster following a disaster, I felt compelled to address this topic.

In February 1993, just two weeks after the first attack on the World Trade Centers in New York, a major east coast snowstorm collapsed the roof of the building housing a datacenter operating the largest automated teller machine (ATM) operation in the U.S. This center was primarily operating ATMs for banks as an outsourced service. The roof collapse caused the building to be condemned and the data center had to be shut down.

This was a professionally run operation with a solid disaster recovery plan.  The thing should have been up and running at a recovery facility within 12 hours and full operations should have been restored within 3 days (in the early 1990s the telecommunications infrastructure used for ATMs required significant time to direct to an alternate processing center). And it could have been back online except that there was "no room at the inn." 

The primary systems running the ATMs were based on Tandem computers, as were most of the systems belonging to the financial companies based in the World Trade Center. The systems displaced by the bombing at the World Trade Center two weeks earlier had been relocated per their business continuity plans to the two recovery sites supporting Tandem equipment at that time (one on the east coast and one on the west coast). These recovery sites were full when the roof collapse occurred. Recovery site contracts are generally first-come first-serve with no guarantee of space when you need it. The presumption is that disasters are rare and the high cost of recovery facilities and equipment is spread across many customers. The specialized computers meant there was limited recovery space and the disaster of two weeks prior filled the available space.

Was this a unique set of circumstances that should not be included in planning efforts? Well the specifics are certainly unique, but the outcomes from back to back disasters and what that meant to getting the ATM processing center back online should not be outside the planning parameters of a solid business continuity plan. The "what-if" scenarios need to include:

* What if the back-up site is unavailable?

* What if staff are affected by the disaster event (the first or subsequent event) and are not available to work on the recovery?

* What if a subsequent disaster knocks out the back-up site before the primary site is back online?

* What if offsite data storage is affected?

* What if transportation is limited during the first hours of the recovery?

These are only some of the questions to ask. The point is to think beyond the initial outage. Katrina evacuees in the Houston area are having to evacuate their temporary shelters as Rita approaches. Recovery plans can be stepped on by subsequent disasters. Recovery efforts from Rita will be more difficult given the resources already committed to Katrina recovery efforts.

As local, state and federal governments and many business are re-evaluating their disaster planning, many IT business continuity plans and the business continuity plans of outsource providers are surely being reviewed. If you are involved in those reviews, be certain to include outsource providers in those reviews. And think beyond a single event scenario. Your backups need backups as you may not be the first in line if and when the time comes.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.

Copyright © 2005 IDG Communications, Inc.