Planning for Disaster - Houston-style, Part 2

A fly on the wall during a disaster recovery operation

Developments of the week in storage

Early Sunday morning late last month, specialty chemical company Champion Technologies failed back its data center to its Houston headquarters and I was a fly-on-the-wall during the process. Originally disabled by power being down in its headquarters building, Champion had to execute a disaster recovery drill and failover its data center operations to a site in Scottsdale, Ariz. That failover occurred Friday evening. All operations, which started at 6 p.m. were completed at 11 p.m. that night - five hours later.

When the power went back on Saturday about 9 p.m., Champion’s IT staff and staff from its storage supplier FalconStor reconvened at the Houston offices, ready to fail back data center operations to Houston in time for the workweek on Monday.

After some preparation, failback started. The uninterruptible power supply, Cisco Catalyst 6509 switch, Cisco MPLS router, Dot Hill storage and Cisco Fibre Channel switches were powered on, according to a carefully-orchestrated plan, documented in Microsoft Project.

To continue reading, register here and become an Insider. You'll get free access to premium content from CIO, Computerworld, CSO, InfoWorld, and Network World. See more Insider content or sign in.

Early Sunday morning late last month, specialty chemical company Champion Technologies failed back its data center to its Houston headquarters and I was a fly-on-the-wall during the process. Originally disabled by power being down in its headquarters building, Champion had to execute a disaster recovery drill and failover its data center operations to a site in Scottsdale, Ariz. That failover occurred Friday evening. All operations, which started at 6 p.m. were completed at 11 p.m. that night - five hours later.

When the power went back on Saturday about 9 p.m., Champion’s IT staff and staff from its storage supplier FalconStor reconvened at the Houston offices, ready to fail back data center operations to Houston in time for the workweek on Monday.

After some preparation, failback started. The uninterruptible power supply, Cisco Catalyst 6509 switch, Cisco MPLS router, Dot Hill storage and Cisco Fibre Channel switches were powered on, according to a carefully-orchestrated plan, documented in Microsoft Project.

FalconStor Network Storage Server appliances, which handle the replication of data from Houston to Scottsdale and vice versa, were powered on to start the replication process back to Houston. Database synchronization from Scottsdale to Houston was finished 6 hours later. At 11 a.m. Champion staff validated the replication and proceeded to assign storage-area network resources to the servers; SAP was brought up and VMware servers were powered on and validated. Microsoft Exchange message stores were mounted. I missed all those operations, blissfully sleeping in a hotel bed two blocks away.

When I arrived at Champion’s Houston office at 11:30 a.m. Sunday, failback was again in full progress. All that was left to do was replicate the deltas back to Houston. Because only a small amount of changes had taken place to the data, this operation took place quickly and without any problems.

After operations had resumed in Houston, users were locked out of the system in Scottsdale, the Scottsdale servers were shut down, SAP was downed and the Exchange message stores were unmounted from the Scottsdale servers. (Compare Data Backup and Replication products)

Everything was ‘All systems go,’ as they say in Texas, by 2:30 p.m. Sunday. Most of us – Stanley Qin, vice president of professional services, excluded – were home in time for dinner. Stanley was on a plane back to FalconStor Software in New York.

Read more about data center in Network World's Data Center section.

Deni Connor is principal analyst for Storage Strategies NOW and host of both the Masters of Storage and Masters of Servers Solution Centers.