Hurricane Katrina disaster recovery lessons still popping up

For at least three days prior to when Hurricane Katrina struck, Marshall Lancaster and his IT team at Lagasse were closely tracking the storm, hoping it would spare his company's New Orleans-based headquarters and data center but preparing for the worst. By the time Katrina made landfall early on a Monday morning in August 2005, Lancaster and his team were in Chicago at the company's backup data center, having already declared a disaster.

At the time, Lancaster was an IT executive with Lagasse, a subsidiary of United Stationers, where he now serves as vice president of IT, Enterprise Infrastructure Services. While Katrina ravaged New Orleans, Lagasse experienced no system down time. In fact, the day after Katrina hit, the company recorded its second-largest sales day, and its third-largest the day after that.

Lancaster related his Katrina experiences in a keynote address at the recent Network World IT Roadmap event in Chicago (register for future IT Roadmap events), which the New Orleans native now calls home. He spoke of the need to consider the people element in disaster planning and how when a disaster strikes, it bears little resemblance to any pre-planned disaster recovery drill. (Read sidebar: "Disaster recovery tips and lessons learned".)

"When an event occurs, it isn't just about whether or not your systems come back online, but where's everybody going to be?" Lancaster said.

Anatomy of a disaster

Lagasse was battle-tested by the time Katrina rolled in, having experienced four hurricanes in the previous few years: Isadore and Lilli in 2002, Ivan in 2004 and Dennis earlier in 2005. Indeed, the company had the drill down pat.

On Thursday, Aug. 25, Lancaster and his team began to take serious note of Katrina by implementing a "Level 1 inclement weather policy," Lancaster said. That basically just tells employees the company is tracking the storm.

The next day, the company went to Level 2, which is when it tells its associates to make sure their homes are in order, with sufficient supplies of food, water and the like. "We were still pretty hopeful [Katrina] was going to veer," he said.

By Saturday morning, Aug. 27, the five computer models Lagasse was tracking all showed the storm pointed at New Orleans. The only question was whether it would be a direct hit. But Katrina was by now so powerful that even a glancing blow was likely to mean substantial damage.

The company declared a Level 3 emergency that morning, which meant planning for the headquarters and data center to be closed on Monday morning. Critical personnel had to be transported to somewhere safe, with access to communications.

Those critical personnel included Lancaster and his IT team, who headed to Chicago to make sure the company's backup systems were ready. "We still had a lot of unfounded optimism that this storm would pass us by and we would be spared," he said.

By that night, with all meteorological models showing Katrina making a direct hit on New Orleans, that optimism was gone. "At 8:55 p.m., we decided to declare a disaster." That means turning on the disaster recovery platforms and using them going forward. By midnight, all Tier 1 applications were online and tested. By 7:33 p.m. the next day, all Tier 2 applications were available. "That means all customer-facing business capacity was online and working."