IT pros share their horror stories

With Halloween approaching I thought it appropriate to share some bewitching IT stories from some of the ghosts (those who want to remain anonymous) and goblins (proud to take credit for their ghoulish tales) belonging to Encompass, HP’s largest enterprise technology user group (I’m the president).

As I quipped with some of our contributors, most of my frightening IT stories involved “me.” The computer just happened to be in the room.

All kidding aside, I do have a couple that drove me to the witches' brew.

In the early days of e-mail the term “viral marketing” meant something different and far more literal than it does now. Our company decided to jump on the mass e-mail broom, and I had instructed our marketing manager to send out our first blast e-mail to our database on a time-sensitive promotion. It simply had to go out. She had been experiencing computer problems that day, but I told her it could wait until after the blast. As you have probably concluded by now, the e-mail had a virus attached and became the unsolicited promo shot heard around the world.

Now everybody has at least one Windozzze story. Back in the mid-'90s, our backend ran on VMS (what’s a blue screen?), and all the desktops had just upgraded (I use the term “upgraded” loosely) to Windows 95. At the same time we released the first version of our software tool, and yours truly was giving a presentation to an oil company in New Orleans. During the presentation I got the blue screen of death four times! Fortunately for me, Microsoft’s Bill Gates had recently made news for getting the BSOD during a demo of Windows 95 at a conference. Ouch.

It’s refreshing to know that I’m not alone. Hold your flashlight under your chin and enjoy the following tales from the IT crypt:

* Pete L. – New York: One time we had a mild hardware problem with one of our ProLiants. The server was extended from the rack and cover open. It was powered up and in production, and we were trying to retrieve a part number or check a fan or whatever. Living up to my reputation as one who loves games, I started tossing a quarter up and down. . . . The quarter fell right onto the motherboard, and the server shut off immediately. The rest was history, and the customer was ticked at us. Funny now; not so funny then.

Steve Davidek – Nevada: One of my staff called me last year while I was at the gym to say that our entire data center had shut down. I immediately (of course) achieved an adrenaline high and rushed back to work. Upon arriving, I found all 20 Windows Servers, our MPE financial system server and our network “shut down”. My tech explained that the UPS was beeping that a battery was low, so he turned the data-center UPS off so it wouldn’t beep anymore. . . .

Anonymous: We needed to add another plug to our police department server room, so the electrician asked if we could shut down our police dispatch server so he could add it. When he finished about five minutes later, the server would not boot. It seems that even though the HP3000 is as rock solid as a VMS server, after running for six years straight without a shutdown the system/motherboard failed. It was exciting to have the dispatch supervisor breathing down my neck while I called HP for help. . . .

Anonymous: About two years ago I arrived at work at about 6 a.m. and had a voice mail from our recreation office saying they could not get on their cash register system (they open for business at 5:30 a.m.). I called their phone number to find out what was going on but got their voice mail and left a message figuring they were with a customer. I did look at their T-1 connection and it was down. So I tried to call again. When one of my staff arrived, I had one of our support techs drive over to the office. There was a sign on the door that they were closed because their power was out. The tech went into the darkened building to find out what was going on and asked who called. The clerk said they had called on their personal cell phone (but didn’t leave that unknown number) at 5:30 and still couldn’t understand why we couldn’t get the network and their PCs working. And how come the UPS next to the cash register was beeping and not letting the PC stay on? Come to find out the power panel had shorted out and almost caused a fire – the panel was as black as if it had burned. The network and phone connections were in the same little room. And it was very dark. . . .

Bill H. – Georgia: This is a story that is frightening by the magnitude of the effect -- deleting a complete directory tree on more than 300 computers with a simple (mistake) change. I was working for the HP Response Center in support engineering, the group that was helping all the engineers in the U.S. taking calls for customers. To help track all the test systems and let all the engineers know where equipment was located, I created an inventory script to document the hardware and software. This was running just fine for months, with some 300-plus systems updating their information every night. I decided it would be useful to store copies of helpful scripts and tools on the systems but picked the wrong directory. When I discovered the problem, I modified the inventory script to remove the files from the wrong directory and store them in the correct one. Unfortunately, due to a spelling error, I removed the entire /usr/local/bin directory and all subdirectories -- on 300-plus machines. I had to send out a very apologetic e-mail to dozens of engineers and learned that all scripts must use the spellcheck option –u.

Richard Buckle – California: Disaster averted. Back in '89 I had brought a team of developers up from Sydney, and we were on the Tandem Cupertino campus when the "big one" hit – an earthquake of 7.1. And even having fallen through false floors and on their sides, the Tandems in the support center kept right on running. Unbelievable.

Buik is president of Encompass, HP’s largest enterprise technology user group and senior vice president of MindIQ, a Norcross, Ga. IT training and eLearning company.