MasterCard factors 9/11 into disaster-recovery plan

MasterCard is gearing to build a back-up and disaster-recovery site designed for the age of terrorism.

O’FALLON, MO. – MasterCard International, which last year opened a gleaming $160 million data-processing campus in Missouri, is gearing up for its next challenge: building a back-up and disaster-recovery site designed for the age of terrorism.

The card-processing giant currently maintains a back-up operation at an unspecified Long Island, N.Y., location, but the Sept. 11 terrorist attacks last year on the World Trade Center spurred MasterCard to rethink its disaster-recovery plans. And a factor it had to take into account is that the Federal Reserve and the Securities and Exchange Commission are pushing for new rules that, among other things, would mandate a two-hour recovery time for financial services companies’ systems.

MasterCard, which processes about $1 trillion annually over its global BankNet VPN, likely will build its new back-up facility about 300 miles from the new data-processing center – the minimum distance the federal government wants to allow, says Artie Ahrens, senior vice president of computer and network operations at MasterCard.

The New York disaster-recovery facility, which is actively involved in a portion of MasterCard’s daily card-processing tasks, was not damaged by the Sept. 11 attacks. But for MasterCard, a new reality became evident. The company’s assumption that it always would be able to fly personnel into this disaster-recovery site if necessary was wrong in that attacks shut down air travel across the country.

“We’re going to move the backup closer, within three to four hours’ driving range,” Ahrens says. “And we’re going to be able to operate it remotely. In the existing facilities in New York, you don’t have all the remote controls. For instance, if I’m starting a job on a mainframe or monitoring a Unix system or reloading software today someone has to sit in front of a console in New York or here.”

New digs

MasterCard is focused on getting its back-up strategy in order, even though the new data-processing facility where some 2,000 IT professionals’ work is built to withstand even earthquakes. The land, in a rural area outside St. Louis, was essentially a donation from a local developer who sold several acres to MasterCard for a nominal fee in order to spur new housing and retail demand around it.

Beneath the quiet hillside, though, there’s a telecom system fit for a metropolis. Southwestern Bell has installed SONET rings and multiple OC-12 links for redundancy. AT&T, MasterCard’s primary carrier and the supplier of its global VPN network, has about 20 employees working full time for MasterCard. Inside one area of the data center, a giant screen, part of the Hewlett-Packard OpenView network management system, displays MasterCard’s traffic volumes across the VPN.

The data center also boasts two IBM mainframes, more than 200 Cisco switches and routers, about 150 servers (mostly from Sun) and 13 EMC systems that pack 200 terabytes of storage capacity.

When it comes to security, MasterCard prefers to maintain its own disaster-recovery site rather than outsource to third-party providers.

“With SunGard or IBM, you have to put dedicated equipment on their floor as you’re renting their floor space,” says Randall Till, vice president of business continuity at MasterCard. “But larger companies are finding that as equipment becomes smaller, you can get a lot of equipment into a smaller space.”

MasterCard has dozens of offices and processing facilities around the world where it could install a disaster-recovery operation. But places where terrorism is seen as on the rise – such as the Middle East – aren’t high on the list for consideration.

At the end of the day, all important card-processing files sent over BankNet, which is tied into bank mainframes and processors around the world, come for final storage at the O’Fallon facility. So it makes the most sense to have the recovery site relatively nearby.

Like most companies, MasterCard’s disaster-recovery procedures – outlined in a manual several hundred pages long that Till presented for a moment’s look – involve a phased-in approach to restoration. Less-important applications might take hours or even days to bring back into use after any significant downtime. What the company calls Tier-1 applications – those involving the most important card-processing data – are backed up continuously through data-mirroring.

Critical data-mirroring

Till says he expects to see the expansion of the use of data-mirroring, which the company performs via Oracle database software and EMC storage equipment. Data-mirroring, or real-time duplication of data, could take hold not just in the new disaster-recovery center that’s envisioned, but at a possible third back-up location under consideration, he says. But he says the process is very involved, and requiring the use of high-speed lines and careful attention to database size.

“When we moved to mirroring, [we had to rewrite] some of the corporate applications we had started to use in the replication process to keep different sites in sync,” Till says.

MasterCard must be in line with any new regulations the Federal Reserve creates for disaster recovery, so the company will target the two-hour restoration time frame the government already has suggested it wants for financial-related information.

“We want to reduce recovery time from 24 hours to two hours,” he says.

The role of MasterCard in the trillion-dollar card-processing industry is critical.

“If we have a member bank that goes down, we do the stand-in authorization processing,” Till says. “And we see that on a regular basis.”