Disaster recovery then and now

Some may see such extensive planning as merely a fire drill in the wake of tragedy, but history bears out the importance of a business continuity plan. Consider Hurricane Andrew, which struck in 1992 and is the worst natural disaster on record. Within two years, 80% of the affected companies that lacked a business continuity plan had failed, according to the Federal Emergency Management Agency.

The events of Sept. 11 have shown companies just how narrow their planning may have been.

"People were not taken into account. Are people still alive? Do they have the mental capacity to work?" says Rich Corcoran, business recovery information manager for Eastman Kodak and one of the most respected IT executives in the continuity-planning field. He has spent 15 years crafting Kodak's continuity plan.

Corcoran says other planning deficiencies were workstation recovery and detailed plans to store and recover vital records, such as those on microfilm or digitized.

Kodak's plan ensures 100 fully functional workstations are available in 24 hours and 300 within three days. And the recovery time for Kodak's enterprise resource planning (ERP) application is four hours for basic availability and 30 hours for total restore. "That is an extremely aggressive . . . plan," he says.

Tom Kelly, senior director of customer services for SunGard, a leading business continuity services firm, says a major deficiency he saw was in the scope of planning. "We've been having a lot of discussions with clients on business dependencies," he says. "Many hadn't thought that through. Peripherals, such as printers, were commonly overlooked."

Setting a new course

Zamba Solutions, a consulting firm, had basic contingency plans such as telephone lists, vendor contact numbers and data backup, but now all that is under review as the company develops a formal written plan.

"The organization as a whole now recognizes that some sort of plan has to be in place. We have a lot of visibility with upper management," says Tom Booth, director of IT. "The work is less on disaster recovery right now and more on business continuity planning."

Booth is helping define 10 to 12 of Zamba's core business processes, such as payroll and benefits coordination, and the internal and external dependencies on those processes that are key to sustaining the business during an emergency.

"It's an eye-opening experience just to do the dependencies for payroll to figure out all that is needed to get that up and running again," he says.

He says the biggest realization is "that this is more than a systems problem."

That realization happened in June for Rich Obrecht, the IT representative for the disaster-recovery team at a leading oil and gas company in Texas, when Tropical Storm Allison paralyzed Houston.

"We've started to form disaster-recovery teams by business unit," Obrecht says. "We've had fairly lengthy interviews with the business unit people to discuss what they need when an emergency hits, when you don't have an office and you've lost paperwork."

The questions are "Where is my engineer, my accountant? How will they get data, tools, mail? We are just realizing how big this animal is," he says.

Pat Parker, director of data systems for CBS Marketwatch.com, says the notion of property loss is changing how the news organization inputs and replicates data between three data centers in New York, Minneapolis and Redwood, Calif.

"The data for our live tickers use to flow into one data center. Now it flows into all three," Parker says.

He says the company's focus is to get everything in writing and form a cohesive back-up and recovery plan that is identical at every data center. "And now we are considering network switching, data routing, replicating domain controllers, equipment needs, electricity and efficient means to restore backup," Parker says. The plan is expected to take six months to complete.

Experts say disaster-recovery plans should evaluate the need for, or the configuration of, replication, redundancy clusters, software change management, remote access, access to back-up tapes or servers, and hot or cold sites. Companies also should establish "quick ship" programs from vendors for product replacement.

Plans also should include remote management in case of biological attack.

"If your site is evacuated, can you manage from a remote site if you can't get to your servers? That takes planning to assure, for instance, that your servers can be booted remotely," Gartner's Scott says.

Testing, testing

Kodak's Corcoran says that without testing you don't have a plan. "At a minimum you have to test once a year for critical applications and infrastructure. Our ERP recovery plan is tested three times a year," he says.

Corcoran says testing must be conducted only in a plan's established recovery center. "That sticky note on the server is of no help at that point," he says.

Chris Leach, national director of technology risk management for Grant Thornton, a global accounting firm, agrees that it is all about testing. "It's sobering when you run your first test and see what you missed. I have never seen a test that did not involve surprises," he says.

He says companies are now asking if backup is enough and if they are doing it right. "We ask them in response, 'Can you pull out your IT department and have it run somewhere else?'" he says.

Cost vs. risk

However, the extent of many organizations' recovery plans eventually will boil down to costs, which can be hard to determine.

Experts say spending can start at $20,000 and go through the roof from there.

Corcoran says cost is hard to quantify because effective continuity planning has to be second nature.

"Any good corporate business continuity plan should be part of the culture," he says. "When you do something in the corporation you have to ask what it changes, if it increases risks. Continuity planning has to be part of the business process model."

But Gartner's Scott says there are low-cost steps that can be taken immediately.

"You can put together a crisis management team that puts flashlights in desks, [and] maintains call lists of personal phone numbers and e-mail addresses," she says.

Teplitsky, the network administrator at the Northeast telephone company who views those costs against the backdrop that use to feature the World Trade Center towers, says his company is considering spending $400,000 just to install redundant storage that would replicate data daily.

Overall, he estimates the company could spend more than $1 million just to ensure up-to-the-day data restoration. And that figure doesn't take into account ongoing maintenance and testing or other parts of the continuity plan.

However, it is those kinds of figures corporate executives will wrestle with in trying to determine how much risk they can endure.