Fed to banks: Improve backup

The Federal Reserve is spearheading a drive to impose strict new disaster-recovery regulations on financial institutions so that trading and banking operations will more quickly rebound from a Sept. 11-type catastrophic event.

WASHINGTON, D.C. – The Federal Reserve is spearheading a drive to impose strict new disaster-recovery regulations on financial institutions so that trading and banking operations can more quickly rebound from a Sept. 11-type catastrophic event.

Financial institutions, while also eager to avert a repeat in any future emergency, say they are concerned these regulations will prove too costly and difficult to implement.

“It’s going to be expensive to do what the Fed is thinking about,” says Paul Hugenberg, IT audit officer at Sky Financial Group, an $11 billion financial services firm.

The Fed acknowledges its ideas on regulating disaster recovery will add costs – how much remains unclear – and even fundamentally change how financial firms organize their central office and back-up operations.

The Fed has summarized its desire for new regulations in a document titled “The Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System.” Comments on it are due today, and banks expect the proposals it contains to become regulations by year-end.

In the wake of the Sept. 11 attacks, Wall Street trading was halted for a week, and the Fed acknowledges that network and data back-up plans proved inadequate, creating a multibillion-dollar payments breakdown among the closely interconnected systems. Federal Reserve Vice Chairman Roger Ferguson is advocating new rules that would require banks, brokerages and other regulated financial firms to be able to resume business within hours in the event of a disaster. Other proposals include requiring financial firms to test back-up systems with their trading partners, and duplicate data center and business operations.

“The events of Sept. 11 graphically demonstrated the interdependence among financial-system participants, wherever located,” Ferguson said during a recent meeting of the Institute of International Bankers in Washington, D.C.

Back-up systems not tested

The Fed discovered that many financial firms in the New York area had not tested their data and telecommunications back-up systems before Sept. 11. Few had planned for the magnitude of the destruction, with offices and telecommunications circuits obliterated. In addition, the commercial “hot site” providers with which the financial firms had contracted were turning customers away because of the demand.

This meant that critical financial information couldn’t be shared electronically, and the domino effect led a multibillion-dollar “liquidity bottleneck” so severe that the Fed was forced to lend large amounts directly to institutions and provide billions more in payments on uncleared checks. The Federal Reserve staff even stepped in to set priorities for the restoration of key telecommunications circuits.

Ferguson declined to discuss the possible new regulations, but public documents and banking insiders provide a clear picture of the direction under way.

In February, Ferguson summoned two dozen of the largest financial firms, including Citigroup, Bear Stearns, Goldman Sachs, Mellon and Merrill Lynch, to confer with the Federal Reserve banks and other regulatory agencies, including the New York State Banking Department and the Securities and Exchange Commission. The Fed, the SEC, the Treasury’s Office of the Comptroller of the Currency and the New York State Banking Department jointly released the draft of the proposed disaster-recovery regulations in late August.

Recovery in two hours

This draft suggests new rules requiring two-hour restoration and recovery, and industrywide testing among banks and their customers. In the document, the Fed suggests that banks prepare for massive telecommuting as backup; a separation of primary and back-up sites by at least 200 miles; and planning for a “split-operations” approach that would duplicate employee and data center installations rather than have a central operation and a secondary backup.

Some see the proposals as daunting.

“The impact of this is huge,” Sky Financial Group’s Hugenberg says. “My bank trades with a bank in Chicago, for instance. The Fed’s white paper says that the bank in Chicago needs to be able to recover. This is the first time we’ve heard we should be testing with outside firms.”

Hugenberg says many in the banking industry expect to see new regulations by year-end, leading to an overhaul of the way banks typically design backup and recovery. Today, it’s often assumed that employees will travel a short distance to a “hot site” or alternate facility, if need be. But the Fed guidelines will promote more geographically dispersed operations in which financial firms would hire employees to duplicate functions done elsewhere. That way, if one office is wiped out, staff in an alternate office would be able to continue operations.

“Before Sept. 11, we didn’t look at our business-recovery plan as it related to human impact,” Hugenberg says. “Now we’re looking at where people are, and will they be too distraught to work? Or simply no longer be around?”

Several Fed documents published since Sept. 11 reveal the banks’ concern that telecom carriers, which are regulated by the Federal Communications Commission, are sometimes the weak link in the disaster-recovery chain.

“Telecommunications vulnerabilities are still seen as a significant source of concern,” the Fed report said. “There is concern that even the telecommunications companies do not have the information they need to provide assurances to financial institutions.

“Many firms believed they had achieved redundancy in their communications systems by making arrangements with multiple telecommunications providers or by contracting for diverse routing, only to discover that all of the lines traveled through any of now well-known single points of failure,” the report said.

A source at Merrill Lynch, who asked not to be identified, said his firm made this painful discovery Sept. 11.

“Verizon was our single point of failure, and we just had no idea this was the case,” he says.

Some carriers, including Verizon and AT&T, contend they have made the effort not only to rebuild destroyed facilities in New York but to reinforce circuit redundancy. AT&T spokeswoman Claudia Jones says AT&T is in discussion with the Fed about its plans.

The Fed wants to organize the banking industry, the 12 Federal Reserve banks and the telecom industry to conduct a series of disaster-recovery tests in the coming months.