How to survive a hurricane without halting business

When Hurricane Katrina swept through Louisiana three years ago, Baton Rouge-based Lamar Advertising was housing all of its IT equipment in just one data center.

Though Lamar suffered just a couple of days of downtime during the storm, IT directors realized they needed a better long-term disaster-recovery and quickly got to work.

"We want to have 99% uptime because we have several hundred offices around the country," says Ed Nettles, vice president and director of IT at Lamar. "[Previously], if we lost power at our primary data center, all of our offices were shut down."

Lamar's offices all connected to the central data center in order to get on the Internet, access e-mail and the various applications needed to run its business of selling billboards, highway logo signs and other outdoor advertising. Before Katrina hit, Nettles and director of network services Peter Dunn were already sketching out a disaster-recovery plan, but the hurricane convinced them to speed up that process.

Lamar rented space in a Tier 4 data center hosted by Network Technology Group in Baton Rouge, and was moving equipment into the facility by April 2006. Lamar also acquired space in a co-location center hosted by Venture Technologies in Jackson, Miss., a location selected because it is hit by fewer natural disasters and is within driving distance.

Lamar rents floor space, power and cooling from the hosting centers, but retains full control over servers and networks. To move data across the two co-location centers and Lamar's headquarters, Lamar selected Unisys SafeGuard 30m software, which allows replication of data and disaster recovery across great distances.

Then came Hurricane Gustav. Hitting Baton Rouge the first week of September this year, it tore the roof off one of Lamar's buildings.

"Gustav came through Baton Rouge and hit us about as hard as you can possibly be hit by a hurricane," Dunn says. "We lost power to this area for about a week, and the rest of the company didn't know…. In all the other states, it was business as usual."

Data for user-facing applications was all replicated to the two co-location centers, so employees outside Baton Rouge weren't affected. In addition to Unisys, Lamar uses a replication engine for VMware's hypervisor, as the IT department has virtualized some of its physical servers.

Lamar suffered a few interruptions to minor applications, but nothing mission-critical. "Not 100% of everything we do is replicated. Internally, we have an amount of acceptable risk with certain types of data," Dunn says.

While test and development is on a lower tier of importance, e-mail, SQL applications and database software are given a high priority.

Branch office employees may not have been affected, but cleaning up after Gustav was no picnic in Baton Rouge. Lamar had recently purchased a second building next door to its main facility, and planned to turn the newly acquired property into its main data center.

Lamar, which uses Cisco switches, nearly 200 Dell servers and 100TB of storage, had started moving data center equipment next door before Gustav hit. (Compare server products.)  Fortunately, Lamar's equipment was on the first floor, because the roof of the three-story building was destroyed. Lamar had been leasing the second floor to another company, which lost most of its computers, but Lamar's own equipment was salvaged.