Lehman Brothers' network survives

Investment giant doesn't miss a beat after IT headquarters is leveled in WTC attack.

By Sharon Gaudin
Network World, 11/26/01

Moments after American Airlines Flight 11 slammed into the Tower One of the World Trade Center, Bob Schwartz, managing director and CTO of Lehman Brothers, looked out his 40th-floor window to see the roofs of nearby buildings on fire and pieces of his own building raining down. In the lobby outside his office, water was shooting out of the elevator banks.

Minutes later, Schwartz and his managers began moving their workers - the core of the global investment bank's IT staff - down the stairwell, away from the fire and chaos, as well as from the network that kept the company running.

As he made his way down the stairwell, the only thing Schwartz had with him to trigger Lehman's disaster plan and alert the company's CIO, working in London that day, was the BlackBerry pager in his jacket pocket.

Disaster recovery: then and now
Security takes center stage
How ready are the nation's networks?
The terrorist network
Disaster recovery on the run: the timeline of events
(requires Flash)
Planning for the worst: Bring in the best
Spending shifts

"The BlackBerries let us e-mail back and forth, and our colleagues in the New Jersey office told us what was going on," says Schwartz, who made it out of the tower 30 minutes before it collapsed. "From the stairwell . . . we decided to trigger the disaster-recovery plan, open up a command center and put voice bridges in place for conference calls."

All but one of Lehman's 625 IS employees, who worked on floors 38 to 40 of Tower One, made it out alive.

And despite the devastation, Lehman's treasury department moved to the back-up recovery site - the day of the attack - and performed its cash-management functions. The day after the attack, the company was trading fixed-income products and had 400 traders online to handle equities when the New York Stock Exchange reopened the next Monday.

"The network was the hero," says Schwartz, who adds that all their preplanning and network buildup paid off. "No information was lost. I'm sure there were a couple little things here and there, but nothing that interfered with our ability to provide service to our customers.''

A network destroyed

Along with the human toll, Lehman suffered a devastating blow to its offices, computer network and infrastructure. Lehman's worldwide headquarters were in Manhattan, with offices in four buildings in the World Trade Center area. The IT department - application developers, engineering teams and the technology management team - was housed in Tower One, and Lehman also had people in three buildings in the World Financial Center next door.

All the desktops, servers and other hardware in Tower One are now rubble, and the other three buildings are unusable as well. In all, the company lost 5,000 desktops, an entire data center and all its networking gear.

Schwartz says the company has not put a figure on the losses the company suffered Sept. 11. But whatever it adds up to be, the financial hit would have been multiplied if the company's lifeblood, its information, had been lost.

"In the planning process, you can never guess what the disaster might be," Schwartz says. "It's always hard to picture what's going to work. We were cautiously optimistic we would be able to restore all the important applications on a timely basis."

Preparing for disaster

"We had completely redundant networks on both sides of the river," Schwartz says, referring to his offices in Manhattan and the back-up site in Jersey City, N.J." Every application that ran in New York also ran in New Jersey. All wide-area links were completely duplicated. We had two lines, one terminating in New York and one terminating in New Jersey. When we lost access to everything in New York, we still had access through New Jersey to all our other 45 branches."

Schwartz says Lehman also had two identical data centers, one in Manhattan and one in New Jersey, linked by fiber-optic cable running under the Hudson River. Backups were automatically performed - one center backing up the other - so each set of backups would be remotely stored from its original site.

"We were prepared for the loss of either one of those two data centers," Schwartz says. "When we were attacked, we were able to increase the amount of bandwidth . . . allowing us to keep all of our other offices up and running. They were able to continue their normal operations as if nothing had happened."

In the days after the attack, the disaster-recovery team began rebuilding the company's IT infrastructure. Vendors such as Compaq pitched in and quickly shipped the hardware and software Lehman needed. In the first three weeks, Compaq had shipped the company 2,500 PCs.

The disaster-recovery team created a trading facility in its former New Jersey data center.

It also worked out a deal with Sheraton Manhattan, taking over the hotel and removing beds and moving in desktops to give people a place to work. The hotel's ballroom became an IT hub with VPN connectivity to New Jersey.

"We were able to almost immediately seat the most critical people to get the business back up and running," says John Manville, manager of networks and telecommunications, and a senior vice president at Lehman Brothers.

Manville says the company is now running on a single data center and working on setting up another one so it can once again have needed redundancy. Today, it is relying somewhat on its London data center, taking advantage of its spare capacity over a wide-area network. "It's giving us a site to back up some of our more critical applications," Manville says.

And Lehman remains committed to keeping its operations in Manhattan. The company announced recently that it was purchasing a 1-million-square-foot office tower on Seventh Avenue to serve as its new headquarters. n