Disaster recovery high on SAN user's list

When Michael Amble's storage-area network is down, it costs the financial-services firm he works for $4 million per hour.

Fidelity National Financial in Jacksonville, Fla., which handles a third of the real estate title insurance issued nationwide, put in a highly available SAN and disaster-recovery (DR) system that ensures the integrity and availability of its business-critical data at a cost of $25 million.

"We believe that if we have an outage today, we could lose up to two minutes of data," Amble says. "That concern justified the expense of putting the SAN in. Outages are just disastrous for us."

In 2004, Amble set out to replace an aging EMC SAN that needed to be managed with a variety of software.

"We were working with [EMC], and when we put this infrastructure together we were looking to have an infrastructure available that we could manage and operate with one set of tools," Amble says. The company chose to centralize all its data on a SAN from Hitachi Data Systems.

"Management just kept saying, 'What if [centralization] doesn't work,'" Amble says. "I said we'd be out of business, and they said, 'We can't let that happen.'"

Amble's SAN consists of two Hitachi TagmaStore Universal Storage Platform USP1100 arrays, in Chicago and Little Rock, Ark. Each attaches to external secondary storage, the Hitachi Thunder 9585V.

The data on the virtualized TagmaStore/ Thunder arrays is replicated and mirrored over an OC-3 link between the two sites synchronously using Oracle DataGuard and Hitachi's TrueCopy replication software. Amble and his staff can manage the information remotely using Hitachi's HiCommand software.

"We have a completely mirrored system," Amble says. "Our problem is that we needed high availability at our central site so we have redundancy there, with the premise that something truly catastrophic has to happen before I'm going to move to a [disaster-recovery] site," Amble says.

Tier 1 of Amble's infrastructure is the TagmaStore storage. Tier 2 consists of the Thunder 9585Vs. Tier 3 is configured as streaming tape backups. Amble mirrors storage from Tier 1 to Tier 2 and then streams it off to tape through silos and sends the tapes off to his disaster-recovery sites.

When the disaster-recovery site isn't in use, Amble doesn't let it sit dormant.

"We utilize a huge amount of space and all the equipment," he says. "We test the DR site regularly and use it for volume and stress testing. Because of the way data is striped on our disks, we just have to move it out of one site, and when we are finished we move it right back."

Amble says most of the time spent bringing up the DR site is used to coordinate the DNS so the network can find it.

At the same time as he is replicating data from Chicago, tapes are being shipped to Little Rock. "If something should happen to the replication, we are just that far away from having the most current set of tapes available," he says.

Using Hitachi's built-in virtualization capability, he migrates data from array to array, where it can be put to the best use.