Diary of a data center move

Neither rain, nor balky backups, nor faulty switch ports keep Aquent's tech team from delivering e-mail service.

A freezing rain drenched the snow-banked streets of Waltham, Mass., as the technical operations team at Aquent, a professional services company, arrived at its data center early on Saturday, Feb. 22, prepared for a moving day that would stretch well into Sunday.

To continue reading, register here and become an Insider. You'll get free access to premium content from CIO, Computerworld, CSO, InfoWorld, and Network World. See more Insider content or sign in.

A freezing rain drenched the snow-banked streets of Waltham, Mass., as the technical operations team at Aquent, a professional services company, arrived at its data center early on Saturday, Feb. 22, prepared for a moving day that would stretch well into Sunday.

This second phase of Aquent's data center consolidation called for transporting e-mail servers from its Boston headquarters and the Waltham site to the collocation space, setting them up and making sure everything worked when users arrived at work on Monday. In the next phase, Web servers will be moved, and in the final phase, an IBM AIX application server.

The first order of business for Peter Lincoln, Tom Ashworth, Kevin Davies and Eric Czerwinski was to back up the 12 servers that were being moved.

After months of planning, the real work began Feb. 6, when the tech team began preparing the collocation space for the incoming servers. During the next two weeks, the team put in analog lines and a T-1 line, and established Internet, frame relay and VPN connections. Next came routers and switches, firewall and load-balancing equipment, plus a Dynamic Host Configuration Protocol server. Routing tables, domain controllers and account databases were synchronized, while directory services and authentication capabilities were extended to the firm's global offices.

CIO Larry Bolick describes moving to a leased space as a "letting-go" for his IT team, which is accustomed to holding a tight rein on its information systems and having open access to its data centers. But Aquent outgrew the cooling and power systems at its Boston facilities, so IT now would have to deal with tighter security and more costly amenities at the new collocation center in nearby Watertown.

Day 1 From left: Dan Perez, one of Aquent's two Application Development Directors, catches Larry Bolick in the hall for a quick discussion of the implications for application/email integration. From left: As Technical Operations Team Leader, Peter Lincoln discusses with Larry Bolick the Challenge du Jour, or, more like, Challenge du Hour, as the countdown to the actual move of email servers begins. Multi-tasking at its best - pen in hand, ear to the phone, and fingers on the mouse, Larry Bolick is in the thick of preparation. All hands are on deck preparing for the move, including the hands of Victor Palma, an intern from Year-Up, an organization supported by Aquent which provides high-tech training for young urban adults in the area. Aquent CIO and weblog author, Larry Bolick, reviews last-minute details with Eric Czerwinski, Senior Windows System Administrator, near "The Cage" at Aquent headquarters. Day 2 Kevin Davies, Senior Systems Analyst, contemplates series of scheduled events prior to the big move day. The work continues at Aquent's Waltham data center with Eric Czerwinski, Senior Windows System Adminstrator, loading the first of two loads for the day. Tom Ashworth, Senior Network Engineer, stays completely focused on the job at hand. Peter Lincoln, Technical Operations Team Leader, leaves nothing unchecked as he readies the initial set of email servers for transport.

Click to see:

By midmorning, 11 of the servers were backed up, and the team started to break down the equipment and pack it up for reassembly in the racks of the data center cage. The final server failed its backup and required a job restart that would take three hours. With the restart rolling, the four staffers loaded servers, rails, cables and tools into their personal vehicles, and drove gingerly through puddles on a pot-holed journey that crawled along at 5 miles per hour on some of the narrow streets.