The new face of disaster recovery

Jason Hamlett was caught by surprise last December when an oil depot explosion destroyed space his company was about to move into and knocked out its existing office and data center nearby while firefighters battled the ferocious blaze. The Buncefield fire burned for several days and was the largest to hit Europe in peacetime.

Fortunately, Hamlett, IT manager for drug manufacturer Fulcrum Pharma Development, was able to keep his business operating using a new means of disaster recovery – a combination of wide-area file services (WAFS) and continuous data protection (CDP).

With offices in Morrisville, N.C., and Tokyo, Hamlett is well aware of the threats posed by hurricanes, earthquakes and other disasters. He and many others have adopted new software or hardware technologies to protect their companies’ business-critical data.

Customers have traditionally deployed remote replication hardware, software or services and tape-based backups with off-site storage to protect their enterprise data and provide business continuity for their organizations. They’re now complementing replication or replacing it entirely with CDP, WAFS or business-continuity appliances or software.

These technologies help them recover data faster and meet their recovery-time objective – the period within which systems, applications or functions must be recovered after an outage. Applications and data critical to the operations of the business, such as transaction-intensive databases, ERP and CRM systems, may need almost instant recovery to keep the business running. Other systems, such as e-mail or less-critical SQL databases, may fit into a secondary recovery scenario.

Reliance on replication

Enterprise-size businesses typically look to replication or mirroring technologies to protect their most business-critical data. Using these technologies, customers can deploy equipment and software in local and remote locations that replicates or saves changes to data off-site, where it is protected and can be recovered in the event of a disaster.

Synchronous replication software, which requires an acknowledgment to each transmission of data and often requires expensive equipment, fits at the top of the data-recovery continuum, where recovery needs to take place in minutes or seconds.

New to the replication market are products from XOSoft and Topio.

WANSync HA and WANSync from XOSoft lets small and midsize businesses replicate their data between local and remote sites.

Topio’s Roadrunner is a replication appliance that costs less than $100,000. In a typical installation, the appliance would sit at a remote location and receive and store data replicated from the primary data center.

Replicating data may not be the first choice for the money-strapped IT manager, because it is costly and complicated to implement. Synchronous replication may be the gold standard for disaster recovery, but there are other technologies that fit the bill depending on a customer’s circumstances.

Among them are CDP software or appliances, data vaulting, managed backup services and snapshot technologies. Newer technologies such as disk-to-disk backup and virtual tape let users get at their data in minutes rather than hours, not in the sometime-in-the next-day-or-two fashion common to tape library-based systems or off-site storage.

Disk replaces tape

While many IT shops say that shipping tapes off-site remains part of their disaster recovery plan, they have taken the recovery portion of the phrase to heart. Because recovering data from tape is time-consuming and costly and because of a series of highly publicized tape losses, IT has increasingly adopted backup technologies that let them save data to disk instead of tape.

With disk-based systems, recovery is much faster than that of tape, and also more reliable. Users no longer have to check and recheck for media errors, and tapes do not have to be shipped off-site to an Iron Mountain or Sungard facility, where it may take as much as two days to get them back in the event of a failure.

As disk-based backup is replacing tape as a backup media, tape is being moved down in the storage food chain. It is being used as an insurance plan should all other measures fail.

Technologies such as virtual tape library (VTL) software and hardware also are taking the place of tape. In a VTL installation, the software backs up data to or secondary disk just as it would to tape, unaware that the emulation is taking place. IT recovers data in the same manner as it would recover data from tape, but because it is installed on disk, they can recover it faster.

Jeff Mery, data center and enterprise storage manager for National Instruments in Austin, Texas, implemented a disk-based backup system from Overland Storage last year.

“Our multivendor backup environment became increasingly more difficult to manage,” Mery says. “At times, people had to come in over the holidays to replace tapes or add new ones to the system.”

Taxed with backing up the crux of National Instruments’ business – the Oracle’s E-Business Suite – Mery became concerned that the company was outgrowing its tape library system. He moved his storage operations to a disk-based backup system, which backs up his servers and expensive primary storage. The system consists of Overland’s REO 4000 disk-based appliance.

By moving to disk, Mery has cut backup times in half and saved time jockeying tapes in and out. After data is backed up to disk, it is moved to an Overland NEO 8000 tape library at Mery’s leisure.

Protecting data continuously

In CDP implementations, data changes are continually saved to disk, and recovery can take place based on an extremely detailed time sequence. Users can simply roll back the recovery of data to seconds, minutes or hours before the event occurred, and because it is a disk-based technology, recovery is fast.

Steve Wilson is a one-man IT shop at Cincinnati Thermal Spray in Ohio. He’s put a new face on data protection to protect his headquarters as well as three remote offices, in Springfield, N.J., Wilmington, N.C., and Houston.

“Symantec’s CPS [Continuous Protection Server] really closes the loop for me,” Wilson says. “I am getting as close to a real-time backup as can be achieved across my wide-area link and all data resides on a single server that I can then back up weekly to tape.”

Wilson uses Symantec’s CPS software at each of four offices to back up 1TB of data. CPS, which installs on a Windows server, backs up data as a series of snapshots to a CPS system and storage located in Cincinnati Thermal Spray’s Cincinnati data center.

Because there are tornadoes in the Midwest, Wilson is working to setup alternative work sites in a number of locations for users in an emergency.

“We snap data at the end of every shift,” Wilson says. “After some experimentation, we found that that was the best fit. We can capture any changes that might have occurred on the shop floor. Engineers can actually go into the Web interface to recover data they may have inadvertently lost.”

While Wilson would not disclose what he paid for the CPS system, he says the savings in manpower alone are enormous. “What used to take an hour or two to restore a single file, now takes about five minutes,” he says.

This same technology has proven to be beneficial for larger businesses such as Baptist Memorial Healthcare in Memphis. Here, Hal Weiss, systems engineer, has put in Revivio’s CPS 1200 to protect his business-critical financial data.

Other companies making CDP products include TimeSpring, Mimosa Software, Microsoft, IBM, HP, EMC and XOSoft. Some CPS software such as Symantec’s and Microsoft’s use snapshots to back up data as often as 64 times a day. Other systems such as Mimosa Software, EMC, HP and TimeSpring back up data continuously and with time-sliding software can reverse a backup to any known good state within minutes of the failure.

Wafting toward WAFS

Fulcrum Pharma’s Hamlett uses Availl Software’s WAFS and CDP software to protect the Windows file servers in his environment.

Last December, when the Buncefield oil depot fire destroyed his company’s new offices, employees were able to access their files and applications without disruption.

As IT staff watched the fire on TV, servers in Hemel Hempstead failed over to servers in the United States and London.

“By having a full local copy of all files in the United State and the United Kingdom – files kept coherent and instantly updated – we kept doing business almost without incident,” Hamlett says.

Availl’s software is installed at each site a customer wishes to recover data from. As users make changes to data, they are synchronized, compressed and transferred over the WAN to a Windows server and storage in the data center. Because of Availl’s acceleration and optimization techniques, access to files appears as if it was over the local network rather than the WAN. In addition to providing disaster recovery benefits, the software assures that Hamlett’s remote offices are always backed up.

WAFS software can vary in cost from $1,000 per server to several thousand dollars per appliance. CDP varies too – from $995 per server to more than $100,000 an appliance.

“If we had relied on only tape backup from the previous nights, we would have lost files and data – certainly not something I’d want to explain to our clients,” Hamlett says. He has been so impressed by WAFS and CDP that he expects to expand its protection to offices in Scotland and France.

Business-continuity appliances

Carl Woody, network engineer for the Dare County School District in Kill Devil Hills, N.C., chose StorServer’s Business Continuity Appliance to protect the data and applications on the file servers in his environment. The StorServer appliance consists of cache memory, disk storage and software tasked with backing up Windows Server 2003 environments.

“The initial problems we had for StorServer were that we had 30-plus servers, each with their own tape drives, each drive with its own tape rotation,” Woody says.

“We may have 10 to 15 tapes per tape drive and then a different person changing the tapes for all those servers, which is very time-consuming, and because people weren’t always familiar with the technology, the backups weren’t consistent and unreliable.”

StorServer removes that responsibility from the teachers and centralizes the backups. Woody put in the StorServer appliance a year ago at an expense of about $40,000.

“Now restorations take about five minutes. Before, we had to find tapes that had successful backups, swap tapes and take 10 to 15 minutes to scan tapes. It took 30 to 45 minutes to locate a file.”

In preparation for last year’s Hurricane season, Jack Rahner, director of IT operations for human-resources outsourcing firm AlphaStaff, deployed MessageOne’s OneSwitch appliance and MessageOne’s managed e-mail and SQL server backup service to provide application continuity for his customers that use Microsoft Exchange. AlphaStaff has offices in Boca Raton, Fla., an area ravaged by Hurricane Wilma last year. Before the hurricane hit, Rahner failed over his Windows applications and replicated data from the company’s Exchange e-mail servers in Boca Raton to the data center in Atlanta, where the company is headquartered.

“We had OneSwitch ready to go before the hurricanes last July,” Rahner says. “We had operational centers in Boca Raton and Atlanta. We can physically transfer people between those sites if there was an issue at one or the other. In support of that, OneSwitch failed-over operations to our Atlanta office as we closed down our Florida office.

“By moving our SQL and Exchange server to MessageOne, we were able to continue operations with minimal lag time even though one of our offices was knocked out,” he says. Wilma brought down AlphaStaff’s Boca Raton office for 12 days.

“We provided payroll for 23,000 people at the time and served 100% of our customers,” Rahner says.

OneSwitch is software that is installed on Microsoft application servers. It replicates application data in real time to a remote backup system over the IP network. Replication status and failover readiness can be monitored by the MessageOne operations center and also through an enterprise management console. When a server fails, the IT manager initiates the failover sequence, which brings up the remote server in as little as 15 minutes. When the primary site is back in working order, the IT manager initiates the failback mechanism.

When David Erwin caught wind of Hurricane Katrina bearing down on New Orleans, he was prepared at least to protect his data, if not his telecom infrastructure.

Erwin, network administrator for law firm Adams & Reese in New Orleans’ Central Business District, uses a managed backup service from MessageOne to protect his Microsoft Exchange databases.

“On Monday morning – when Katrina hit New Orleans – the motion detectors in our data center went off because the building was shaking so much. We lost power and at 6 a.m. we failed over to a MessageOne data center in Chicago and had complete e-mail and BlackBerry continuity of our relocated users in Baton Rouge 15 minutes later.”

At Adams and Reese, business is continuing as usual thanks to MessageOne’s e-mail failover service. The company, which evacuated its offices to Baton Rouge, is finally back home.