Flood experience led to better disaster planning ahead of Hurricane Ike

* After flooding from Tropical Storm Allison, NetIQ developed a better disaster recovery/business continuity plan

After flooding from Tropical Storm Allison knocked NetIQ's data center out of commission for a week in 2001, the company developed a better disaster recovery/business continuity plan. This September, Hurricane Ike forced the plan into action. Read on to see how the company fared when Ike's winds and water slammed southeast Texas with a devastating blow.

In 2001, Tropical Storm Allison stalled over southeast Texas and dropped more than 35 inches of rain in the state over a span of a week. The city of Houston experienced unprecedented flooding all over the city, but especially hard hit were the major business and medical districts. In addition, virtually every major roadway was impassible due to high water.

Software company NetIQ fell victim to Allison’s flood waters when the bottom floor of the company’s office building was submerged and electrical equipment was destroyed. This left the building – and NetIQ – without power for a week. Though the company brought in generators that would power some office equipment such as phones, NetIQ was forced to limp along in a reduced capacity while still trying to support business operations. And with the widespread street flooding, employees couldn’t get to the office to do their jobs.

Having gone through such a bad time during Allison, NetIQ’s vice president of IT Dohsung Yum vowed to have a strong disaster recovery/business continuity plan in place for the next emergency. Hurricane Rita in 2005 – which barely brushed Houston – gave him a chance to practice the plan, but Hurricane Ike in 2008 proved to be the real test. On Sept. 13, Ike knocked out power to more than 2.6 million Houston area businesses and residences, including the building NetIQ occupies. This time, says Yum, his company was prepared.

Yum’s first decision after Allison’s flood waters receded was to get out of running his own data center. He looked for a suitable company to host and manage NetIQ’s production servers, and he found CyrusOne just a few miles down the road from his own building. CyrusOne offers managed hosting, colocation and managed IT services in its 81,000 square foot Houston data center. NetIQ moved all its production servers to the CyrusOne facility and now Yum doesn’t worry about keeping his data center up and running anymore.

Having covered the data center, Yum pulled together a disaster recovery/business continuity team to conduct a business impact analysis. This helped NetIQ understand the key functions and applications that are a high priority to preserve or restore quickly during a disaster. The team based its disaster recovery/business continuity plan on the likely risks to each of its locations. For instance, hurricanes are an annual threat to the Houston office, but certainly not a concern for the Seattle office. However, whether it’s from a hurricane or a snowstorm, it’s possible for any business facility – regardless of location – to lose power for an extended period of time. So the team came up with ways to cope with loss of power.

The team also learned to leverage NetIQ’s multiple sites to provide continuous access to customer support. When the Houston office couldn’t cover support, the duties were temporary handed off to another of NetIQ’s offices.

Yum says it can be a struggle to get people to think about disaster recovery/business continuity planning throughout the year when there is no disaster. It’s essential, however, to keep the business people mentally prepared, and regularly testing planned procedures is one way to do this. For example, the company tests switching the phone system over to run via the CyrusOne facility or elsewhere.

So how did NetIQ cope during the onerous Hurricane Ike? Very well, actually. Hours before power was lost to the storm, CyrusOne transitioned to generator power to ensure continuous operation of the servers in its care. CyrusOne informed NetIQ ahead of time that it would be doing this. CyrusOne also provided workspace for up to 30 NetIQ staffers in case they needed a safe place to work. As far as the information systems were concerned, there was hardly an interruption at all - even as millions of Houstonians went a week or more without power.

Yum admits that one weak point in the NetIQ disaster recovery/business continuity plan was redundancy of key personnel. A strategic member of the customer support team lives in a suburb of Houston that was hit especially hard by Ike. He had to turn his attention to his own family’s well being, which took his attention away from NetIQ for a week or two. For the next disaster, Yum says he’ll have the people aspect covered better.

For anyone concerned about disaster recovery and business continuity, Computerworld offers a set of best practices to keep the business running as smoothly as possible. Let’s hope you never need to put your disaster recovery measures to use. (Compare Data Backup and Replication products)

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Related:
Take IDG’s 2020 IT Salary Survey: You’ll provide important data and have a chance to win $500.