The data center from hell, Part 1

Opinion
Sep 23, 20086 mins

* The realization of horror

Seen any good horror movies lately? Here’s the script for a security geek’s version of the classic slasher flick.

Jan Buitron, CISSP, MCSE, ITIL Foundations Certified, Network + is one of our many gifted graduate students in the Master of Science in Information Assurance (MSIA) program at Norwich University. Because of her extensive experience in the network administration and security fields, she offered to work with me throughout her program by keeping a learning diary which is providing all of us in the MSIA team with a great deal of insight into the strengths and weaknesses of many aspects of our program in line with our philosophy of continuous process improvement.

Jan recently showed me a hilarious report on some of her experiences at one of her previous workplaces. She has kindly allowed me to publish it for your amusement and edification. The remainder of today’s article is entirely Jan’s with minor editorial changes.

* * *

Introduction

Years ago, I worked in the IT department of a small manufacturing company in Colorado while finishing my bachelor’s degree in computer information systems. This was prior to becoming an information assurance professional. Even at that time, it was obvious that the company’s infrastructure protection weakened its security.

Location

The company’s central data center was well camouflaged in an inconspicuous location in a moderately sized city, but that seems to have been its only advantage. The facility sat in an older, heavily industrialized part of the city. The streets were in need of constant repair because of the 18-wheeler trucks passing through the area; the trucks also created noise and vibration.

The condition of the streets was so bad that several employees of the facility had to repair the suspension systems on the vehicles they used to commute to work. The city frequently tore up the roadways to fix deteriorating underground pipes and other aging infrastructure, often hampering employee access to the site. Just as employee access to the location was hindered, it could also happen that access by emergency vehicles such as ambulances or fire trucks could have been hindered. The site was also a product showroom, and the poor condition of the infrastructure around the site could conceivably reduce customer visits.

The parking lot for the leased facility was also in bad condition: the concrete parking barriers each had long pieces of rebar protruding from the top surface. One day I pulled my car too far over a piece of the rebar and the rebar hooked into a plastic air scoop on the bottom of my car’s engine and completely pulled it off when I backed up to leave for the day. The result was that the car overheated and had to have an $800 repair to fix a cracked radiator and install a new air scoop.

The wrong side of the tracks

Very close by on the north side, there were railroad tracks which were continually used to park railroad cars marked “Flammable” and “Capacity, 33,000 gallons.” At one point there were 35 such tanker cars within 150 feet of the building or closer. I had at least one nightmare that a tanker exploded while I was at work. In addition, two sets of the tracks were actively used by trains, causing frequent noise and vibrations.

On the south side of the data center, about 250 feet away, was a major, raised six-lane highway adding to the noise and vibration. There was even an incident in which a passing motorist fired a high-powered rifle at the roof air conditioner on the building and disabled it. That air conditioning unit was the only unit which cooled the company’s server room; unfortunately, there was no backup cooling unit. It was end-of month processing, so the data center manager had industrial-sized fans brought in to cool the server room.

Finally, there was an oil refinery located less than three-quarters of a mile away to the northwest of the site.

Neighborhood risks

There was no residential population base; the individuals who regularly traveled through the area were mainly employees of the local businesses or homeless persons. Once, there was a knife fight on a nearby corner around 9 a.m. and two police cars were on the scene to manage the occurrence.

Natural risks: storms

One day the rainfall increased to 3 inches per hour. To my dismay, other workers at the site began to describe the site’s “yearly floods!”

Fortunately, the facility was raised several feet above the level of the street and was accessed by an eight-foot-high stairwell. One of the facility employees explained, or perhaps bragged, how the water immediately outside the facility was up to his chest when he ventured to walk down the stairs during an inundation in one of the previous years.

Since the one storm drain in the street in front of the facility was inadequate to cope with the heavy downpour, we watched as the street in between the facility and the six-lane highway began to fill with water. After about half an hour, the water overtopped the street curbs. Cars attempting to drive on the road began to flood out or just plain float. Only 18-wheelers were able to drive down the flooded street with impunity.

Then, as the water began to encroach into the company’s parking lot, the facilities manager allowed the company employees to drive their cars up a ramp into the manufacturing area. This was a good thing, as the water was halfway up most cars’ hubcaps by this time. Had the water risen another three feet above its crest, it would have reached the same level as the data center.

Fire and earthquakes

There were no fire drills at the company during the fifteen months I worked there, despite the fact that the company’s manufacturing materials were flammable. There was a fire hydrant conveniently located right next to the facility, which was one of the few positive features of the location. There had been a magnitude-6.7 earthquake in the city more than 35 years before, but the company had no earthquake plans.

Other hazards

Problems from bad infrastructure permeated everything. One day in late morning a backhoe working to repair a street struck the power pole that carried power to our facility. We immediately called the power company to repair the power feed, but they said they could not come out until the next day. Our battery backup only lasted for five hours. To their credit, the company increased their battery backup system so it could keep the servers running for 18 hours.

* * *

In the next installment of this three-part series, Jan discovers that yes, it can get worse.