Nine network pros give first-person accounts of working through tough situations.
Mike Taylor: CIO, Todd Pacific Shipyards, Seattle.
Walking into disaster the first day on the job
|
Mike Taylor CIO, Todd Pacific Shipyards, Seattle
It was the summer of 1998. I interviewed [for the CIO position] at Todd Pacific Shipyards, and it seemed like a good company, although I knew from working in Seattle that the IT department was not leading edge.
The night of the first day I joined, we did a backup of the homegrown ERP system, which was a SQL 6 application. The next day I came in and the system wouldn’t come up. Worse, although they’d done back-up tapes every night, no one had ever restored and verified them. From the 10 months prior, we didn’t have any data that we could use; the back-up tapes were useless. And we were told that a corrupted SQL 6 database was unrecoverable. We didn’t know what we were going to do.
In the meantime, I was getting calls from two or three other firms I had interviewed with, and I was saying to myself, “What should I do?” However, I didn’t take another interview; I felt like I had committed myself to this job.
For the next 30 days, we were off the air, which included the systems that handled inventory, purchasing, accounts payable and project costing. Folks were using paper and pencil, and if checks had to go out, accounting would use a typewriter.
It was a situation where we worked every day, hitting the phones, just following the recovery trail. Someone would give you a name, you’d call him and he couldn’t help, but he’d give you another name. The Big 5 firms were no help, and neither were the three largest recovery firms.
We finally found someone at a company in San Diego recommended by a lead we called in San Francisco. He said he would try to get the data off the disk for $5,000 per day, but if he couldn’t he wouldn’t charge anything. We took the database disk out of our server and sent it with our best software person to San Diego. Within three days, they got the data off our disk and formatted for delivery to Microsoft. I don’t know why he charged $5,000 per day, but he recovered all but 219 out of 2.2 million records. The next phase was feeding the raw data into a new database. A Microsoft contact gave me someone at headquarters who was kind enough to put me in touch with a manager at the R&D center in Irving, Texas. They were able, over a five-day period, to develop many algorithms to get the database pointers straightened out. I sent them some Made in Washington gift baskets, besides paying the bill, as an incentive and a thank you, because we were sitting on our hands. We were on the cusp of having to report our situation to the U.S. Securities and Exchange Commission.
Being the director of IS, you can’t give up on a project. If one door closes, you have to look for other ways to get this thing solved. That’s a trait that’s learned but you also have to have a certain doggedness in your personality. Shortly after the database was again operational, I implemented the best disaster recovery plan in Seattle!
– As told to Mary Brandel
Previous account | Next account: Pushing through a new and misunderstood technology
