A data center story for the ages: the fuel bucket brigade

The effort to save a NYC data center following Hurricane Sandy is remembered in a new film

The improbable story of the data center saved by a fuel bucket brigade is not told with fondness, even if there is a sense of pride and humor in its retelling.

Those involved recall the stink of diesel fuel in dark stairwells illuminated by flashlights. There was the sheer physical exhaustion of carrying fuel up 17 floors to deposit into a rooftop generator. There was the sleeping on floors and the lack of showers, and the unrelenting stress that their efforts could suddenly unravel and fail.

The Peer 1 bucket brigade carries fuel to the rooftop generator

Then there was the broader weight on all involved about the impact that Hurricane Sandy was also having on their friends, family and on New York City itself.

The effort to save the data center during and after the October 2012 storm is captured in a documentary-style film that Peer 1, the data center owner, produced and showed Tuesday to a group of customers and employees.

The film, with photos of data center rescue efforts along with interviews of participants, doesn't attempt to romanticize or triumph the events. It just lays them out, and leaves the viewer with a sense of the sheer exhaustion watching the effort of the participants and the constant uncertainty of the outcome.

The idea of carrying fuel to the roof "seemed like a ridiculous idea," said Michael Pryor, president of Fog Creek Software, a large user of the Peer 1 data center, in an interview. "It just didn't see feasible."

Pryor's idea, retold during the film, was to bring some extra pumps from fish tank in his office to the nearby Peer 1 data center to see if they might help get fuel to rooftop generator. The proposal drew healthy laughter from the audience, something Pryor now chuckles at as well.

But his thoughts of using fish tank pumps helps explain how the data center workers and customers, in the early going, were grappling to come up with ideas for keeping a roof top generator that was burning 40 gallons of fuel an hour out of a 200 gallon capacity tank running.

Fog Creek makes a software management platform used by developers, and the loss of operations at the 5,000 square-foot data center at 75 Broad St. would have hurt its customers and its ability to generate revenue.

Michael Mazzei, Peer 1 data center manager (Photo: Patrick Thibodeau/Computerworld)

Peer 1 was a short walk from the Fog Creek offices, and as the company grew so did its use of the data center. The data center provides Fog Creek its network and infrastructure for customer equipment.

There's an area set aside for Peer 1 customers to work on their systems.

As flood waters entered the lobby, Jeffery Burns, a worker in the data center, told of hearing the sound of a waterfall in the elevator shaft that eventually flooded the basement and disalbed the fuel pumping systems.

"Everybody in Manhattan has the same problem, how to relocate infrastructure that - for the last 50 years - has safely lived in basements," said Michael Mazzei, manager of the Peer 1 Broad Street data center.

Despite the damage caused by Hurricane Sandy, data centers won't be leaving Manhattan anytime soon. The island is one of the most networked places in the world.

The Peer 1 data center is in the former International Telephone & Telegraph headquarters building and has multiple network connections that give customers quick access and low latency. It's also a short walk to the waterways.

NYC Mayor Michael Bloomberg this month released a multi-billion dollar, multi-decade plan to protect the city from powerful storms, and the impact of climate change. The plan includes the construction of barriers in low-lying parts of the city.

Bloomberg's plan has broad IT and telecommunications implications.

It seeks a higher level of reliability and service for telecommunication facilities. The specific recommendations include requiring new hospitals "to increase their IT and telecommunications resiliency by installing two independent points-of-entry for telecom and communication to reduce the risk of outages from a single supplier."

But for a business such as Fog Creek, the hurricane exposed its lack of a back-up plan. The company has been working to fix that weakness in the months since.

For instance, Fog Creek has improved its data replication capabilities and is developing an ability to switch operations, in an emergency, to a separate facility. One tool now runs in Amazon's cloud service, though that migration has been planned prior to the storm.

The storm made clear to Fog Creek that it hadn't paid enough attention to the possibilities of bad things, like a massive storm, happening, said Pryor. That was a positive outcome of the experience, "because it did kick us in the butt and get us to fix a lot of things that were broken."

In his office, Mazzei points to the part of the floor he slept on -- next to his phone.

The bucket brigade wasn't the first move by the group. Initially, they tried moving 55-gallon drums up 17 flights of stairs, but the physical exertion proved too much.

Gradually, though, a realistic plan came into place. The team acquired supplies, pumps, hoses and buckets from Home Depot and other places. Organization, rules and efficiency arrived.

One rule: "Go slowly, work safely, always have your flashlight," said Mazzei.

For 72 hours, the effort continued and the data center was able to remain in operation until a more stable fuel supply was established. For the future, one option the building owners are giving consideration to is putting submersible pumps that could continue operating in a flood, said Mazzie.

There were a lot of motivations at work in manning the bucket brigade. For customers, it was about keeping their businesses up. For the data center workers, it was about not failing.

No one is really taking credit for saving the data center.

Mazzei is very matter-of-fact about the effort, and describes it more as an organic process that developed on its own as ideas and processes emerged collectively.

His co-worker, Burns, a film editor who works part-time at the data center, said something particularly flattering about Mazzei. "It was just really important to not let him down, not because he was going to get upset, but because he never let us down," said Burns.

Patrick Thibodeau covers SaaS and enterprise applications, outsourcing, government IT policies, data centers and IT workforce issues for Computerworld. Follow Patrick on Twitter at @DCgov, or subscribe to Patrick's RSS feed . His email address is pthibodeau@computerworld.com.

Read more about data center in Computerworld's Data Center Topic Center.

This story, "A data center story for the ages: the fuel bucket brigade" was originally published by Computerworld.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Must read: Hidden Cause of Slow Internet and how to fix it
Notice to our Readers
We're now using social media to take your comments and feedback. Learn more about this here.