Hurricane Katrina disaster recovery lessons still popping up
Tiering applications, paying attention to people issues are key
By Paul Desmond
,
Network World
, 05/06/2008
- Share/Email
- Tweet This
- Print
For at least three days prior to when Hurricane Katrina struck, Marshall Lancaster and his IT team at Lagasse were closely
tracking the storm, hoping it would spare his company's New Orleans-based headquarters and data center but preparing for the
worst. By the time Katrina made landfall early on a Monday morning in August 2005, Lancaster and his team were in Chicago
at the company's backup data center, having already declared a disaster.
At the time, Lancaster was an IT executive with Lagasse, a subsidiary of United Stationers, where he now serves as vice president
of IT, Enterprise Infrastructure Services. While Katrina ravaged New Orleans, Lagasse experienced no system down time. In
fact, the day after Katrina hit, the company recorded its second-largest sales day, and its third-largest the day after that.
Lancaster related his Katrina experiences in a keynote address at the recent Network World IT Roadmap event in Chicago (register for future IT Roadmap events), which the New Orleans native now calls home. He spoke of the need to consider the people element in disaster planning and
how when a disaster strikes, it bears little resemblance to any pre-planned disaster recovery drill. (Read sidebar: "Disaster recovery tips and lessons learned".)
"When an event occurs, it isn't just about whether or not your systems come back online, but where's everybody going to be?"
Lancaster said.
Anatomy of a disaster
Lagasse was battle-tested by the time Katrina rolled in, having experienced four hurricanes in the previous few years: Isadore
and Lilli in 2002, Ivan in 2004 and Dennis earlier in 2005. Indeed, the company had the drill down pat.
On Thursday, Aug. 25, Lancaster and his team began to take serious note of Katrina by implementing a "Level 1 inclement weather
policy," Lancaster said. That basically just tells employees the company is tracking the storm.
The next day, the company went to Level 2, which is when it tells its associates to make sure their homes are in order, with
sufficient supplies of food, water and the like. "We were still pretty hopeful [Katrina] was going to veer," he said.
By Saturday morning, Aug. 27, the five computer models Lagasse was tracking all showed the storm pointed at New Orleans. The
only question was whether it would be a direct hit. But Katrina was by now so powerful that even a glancing blow was likely
to mean substantial damage.
The company declared a Level 3 emergency that morning, which meant planning for the headquarters and data center to be closed
on Monday morning. Critical personnel had to be transported to somewhere safe, with access to communications.
Those critical personnel included Lancaster and his IT team, who headed to Chicago to make sure the company's backup systems
were ready. "We still had a lot of unfounded optimism that this storm would pass us by and we would be spared," he said.
By that night, with all meteorological models showing Katrina making a direct hit on New Orleans, that optimism was gone.
"At 8:55 p.m., we decided to declare a disaster." That means turning on the disaster recovery platforms and using them going
forward. By midnight, all Tier 1 applications were online and tested. By 7:33 p.m. the next day, all Tier 2 applications were available. "That means all customer-facing business capacity was online and working."
Partner Content
www.bmc.com
Gartner 2009 Magic Quadrant for Job Scheduling
Gartner has positioned BMC CONTROL-M in the Leaders Quadrant of their "2009 Magic Quadrant for Job Scheduling." The report assesses the ability to execute and completeness of vision of key vendors in the marketplace. Read a full copy today, courtesy of BMC Software.
Download whitepaper
Dell's SMART Approach to Workload Automation
Read a compelling case study by EMA, Inc. to learn how Dell uses BMC CONTROL-M to cut cost and increase productivity with workload automation.
Download whitepaper
Workload Automation Cost Savings 2 Minute Video
A major computer manufacturer uses BMC CONTROL-M and just four people to schedule and run over 85,000 jobs every month. By switching to BMC CONTROL-M, they more than quadrupled the workload without adding a single staff member. See how in this 2-minute video overview.
Go to video
Comment