Katrina exposed flaws in original disaster-recovery plans.
The New Orleans law firm of Chaffe McCall found two major flaws with its disaster-recovery plan. First, the firm had another office in Baton Rouge where employees could go to work, and it had backup tapes that contained its data, but that office had no tape drive to restore tapes from, says James Zeller, senior network manager for the firm.
Second, the firm had consolidated its Internet traffic so the Baton Rouge office connected via the firm’s T-1 WAN to the New Orleans office, and then to the Internet via the New Orleans office’s dual Internet T-1s. With all the New Orleans network services down, the firm had no Internet access. The New Orleans office housed the only e-mail server the firm had.
Rewiring New Orleans: part one with link to slideshow
Today, the firm has T-1 Internet access for the Baton Rouge office to supplement the connections in New Orleans. That way a hurricane that cuts off the New Orleans office will not affect Internet access for workers in the Baton Rouge office. This is important, because after Katrina, the Baton Rouge office was temporarily expanded to accommodate about 40 extra employees who needed the Internet and e-mail to do business. The firm also set up offices in Lafayette, La., and Houston from which workers had access to applications and data via Citrix software on company-issued laptops.
To address data restoration, the company has bought Xosoft replication software that can fail one site over to the other as necessary in a disaster. While primary copies of files may reside in either Baton Rouge or New Orleans still, they are constantly being replicated so each site has a complete set of the latest versions of files. Tapes and tape drives are no longer necessary to create the complete data store at either site.
In the event of a storm threatening New Orleans, 12 hours before predicted landfall, the New Orleans office would be shut down and the Baton Rouge office would become the primary site. In the event the storm also hit Baton Rouge, servers there could be shut down entirely until power came back on, but data would be preserved. “We’re not concerned about having to shut down entirely for awhile,” Zeller says. “The nature of our business doesn’t require us to be up 24/7 while the hurricane is blowing over.”
During Katrina, no one could get into the firm’s New Orleans 1100 Poydras St. office near the Superdome for days. In the meantime, Zeller and his fellow IT staff tried to acquire hardware to run the tape backups, but ran into problems. “The unexpected surprise was the logistics of shipping things. We couldn’t get things shipped even to Baton Rouge,” he says. That meant the company couldn’t set up its data center. “That office wasn’t ready to be a disaster-recovery site of that magnitude,” he says.
Finally, workers made it into New Orleans and walked up the 23 flights of stairs to Chaffe McCall offices, broke through a wall and carried down servers in the dark to bring back to Baton Rouge.
That revealed a flaw in the law firm’s disaster planning. “We had our data, and it was safe in New Orleans, but it was inaccessible,” he says. “We were protecting our data, but we weren’t fully protecting the processes of our business.”
Zeller has set up a Web-based e-mail system to forward mail so during a blackout when both Baton Rouge and New Orleans are shut down, e-mail would be available through a separate Web site domain.
Also, the law firm has set up a Web site where employees can log their current location in an emergency. The site also contains phone numbers of close friends and relatives who live out of the likely path of hurricanes who will know the locations of employees.
Chaffe McCall has learned that the best disaster-recovery planning can’t bring the business back up any faster than the people who work there can learn to cope. “That first week a lot of people were still getting their acts together on a personal level realizing they had just lost everything they owned,” Zeller says.
