Neither rain, nor balky backups, nor faulty switch ports keep Aquent's tech team from delivering e-mail service.
A freezing rain drenched the snow-banked streets of Waltham, Mass., as the technical operations team at Aquent, a professional services company, arrived at its data center early on Saturday, Feb. 22, prepared for a moving day that would stretch well into Sunday.
Keep up with the move in Larry Bolick’s exclusive Weblog.
This second phase of Aquent’s data center consolidation called for transporting e-mail servers from its Boston headquarters and the Waltham site to the collocation space, setting them up and making sure everything worked when users arrived at work on Monday. In the next phase, Web servers will be moved, and in the final phase, an IBM AIX application server.
The first order of business for Peter Lincoln, Tom Ashworth, Kevin Davies and Eric Czerwinski was to back up the 12 servers that were being moved.
After months of planning, the real work began Feb. 6, when the tech team began preparing the collocation space for the incoming servers. During the next two weeks, the team put in analog lines and a T-1 line, and established Internet, frame relay and VPN connections. Next came routers and switches, firewall and load-balancing equipment, plus a Dynamic Host Configuration Protocol server. Routing tables, domain controllers and account databases were synchronized, while directory services and authentication capabilities were extended to the firm’s global offices.
CIO Larry Bolick describes moving to a leased space as a “letting-go” for his IT team, which is accustomed to holding a tight rein on its information systems and having open access to its data centers. But Aquent outgrew the cooling and power systems at its Boston facilities, so IT now would have to deal with tighter security and more costly amenities at the new collocation center in nearby Watertown.
Day 1
Day 2
|
By midmorning, 11 of the servers were backed up, and the team started to break down the equipment and pack it up for reassembly in the racks of the data center cage. The final server failed its backup and required a job restart that would take three hours. With the restart rolling, the four staffers loaded servers, rails, cables and tools into their personal vehicles, and drove gingerly through puddles on a pot-holed journey that crawled along at 5 miles per hour on some of the narrow streets.
Who are you?
The procession arrived at noon, and the team handed over photo identification to the security guard in exchange for the cage key. They signed in, clipped on badges and proceeded one by one through the “mantrap,” a glassed security enclosure that looked like a Star Trek transporter room. Finger scans and a floor scale authenticated each person’s prints and weight, and personal access codes were typed into a reader device.
Once cleared, video cameras recorded the staff’s progress, while the facility’s operations manager oversaw the foursome as they unloaded and stacked servers and equipment bins at the loading dock onto a pallet cart and dolly.
By 12:30 p.m., they were rolling through a long, white hallway one staffer described as reminiscent of the movie “Dead Man Walking.” On to an elevator draped with moving pads, then to the 8-by-12-foot cage space. The carts stopped outside a sliding gate where five black steel racks stood side by side, centered about two feet inside the cage fence. A constant flow of air roared overhead.
There’s a level of trust in a collocation space that takes getting used to, Lincoln says. He says you have to get beyond the fear that someone could reach in and unplug something. “You never know who your cage mate is, and it’s a bit unsettling to go home at night and not worry about your assets,” he says.
Rack and roll
Blue, green, yellow and white cables adorned the Aquent racks that house previously installed gear, including a Bay Networks 4000 Extranet switch, three Cisco 3700 routers and a Sun Ultra 10 tower running the firewall application.
Two new 1U Dell PowerEdge 1650 servers also were racked, and one had been queuing the firm’s e-mail since 10:30 p.m. the night before. At the bottom of each rack, power control units house 40 jacks, while Belkin keyboard-video-mouse (KVM) switches sit atop the three middle racks. Two Dell monitors and keyboards occupy the second and fourth racks midway down.
The men worked in teams of two, hefting, balancing and installing the servers according to the schematic pulled from Lincoln’s case, at times joking about the potential for scraping their heads on the rails that protrude from the racks. The first and only injury was a small cut to Davies’ finger from a sharp rack edge.
The servers slid easily onto rails that Czerwinski had placed days beforehand. While some rail adjustments were required, the remaining servers were measured by eye for the correct rack slot, and the screws were placed by hand and drilled with a cordless tool.
Figuring out the rack layout took several iterations, Bolick says. Server placement was largely dependent on which servers would be changed most often. “We put the older servers on top. We didn’t want the most critical servers near the cage door to be at risk of bumping into them every time you walked by,” he says.
In less than an hour, all 11 servers fit snugly in racks, including a Loader Express Library Tape System, OptiPlex Gx1 servers, Dell PowerEdge servers – 2300s, a 2400, a 2450 and even the 7U-high Dell PowerEdge 6300, which proved the trickiest lift-and-slide because of its size and weight. Plugging his laptop into a LAN connection, Lincoln fired up pcAnywhere to check the status of the back-up job on the remaining Dell 6300 server in Waltham, and, satisfied, he rode the empty cart like a skateboard, leading the team out to get lunch at a nearby pizza shop.
Tangled up in blue
By 2:45 p.m., Davies and Czerwinski passed back through security and started cabling-up the network, power and KVM connections to each server, while Ashworth and Lincoln rode back to Waltham to get the remaining e-mail server. An hour later, the Dell 6300 was wheeled to the cage door, and with rack rails screwed in, all four staffers lifted and slid it into place.
Like a surgeon wielding a screwdriver and wire cutters, Ashworth worked at removing a screw that had jammed into the back of a 2300 box, as he attempted to install a cable management tray that wraps cables accordion-style at the back of the server. This would provide untangled access for the box to slide in and out. Before long, the cable tray was scrapped when the team realized that there weren’t enough available screws for installation. However, they left plenty of slack cable so tray units could be added at a later date.
The team continued the arduous assembly of cable connections: Green Ethernet cables are for trusted servers, yellow for the demilitarized zone and black for the untrusted servers. They all link back to the Cisco switches that divide the network into zones.
Blue cables connect servers to the routers and firewall. Power cords are plugged into the back of each server, with staffers weaving the cords down under the bottom-most server to plug into the power unit at the base of each rack. In some instances, a two-foot step stool was needed to weave server cables up through racks to reach the KVM switch, or over the tops of racks to wind cables down for plugging into the monitor and keyboard.
As the untangling, plugging and weaving progressed, Davies used a label maker to tag the front and rear of each server, its cables and corresponding switch ports. The cable work wound down and the final connections on a 6300 occurred at about 7:30 p.m. Lincoln added the final touches, tucking all the cables uniformly into the grooves of the rack casings. “Working with cables is the hardest part. You have to take the time to do it right because you only have the system down once,” he says.
We are the champions
Two hours before the cable work was complete, Davies fired up the KVM switch, monitor and keyboard, and proceeded to telnet into the Cisco switches, assigning ports to ensure that traffic would get routed to the correct virtual LAN. Likewise, Ashworth used the second monitor and keyboard to configure TCP/IP ports.
Once the configurations were set, power was turned on and network connections were tested, beginning with the IRIS/Chat server that runs the Intranet and Help Desk collaboration software. Then they powered up the e-mail servers, one at a time.
Powering up is tricky, Czerwinski says. It needs to be sequential because the servers have been running in a steady state for two years, and there’s no way to know ahead of time if memory has been corrupted or if something was jarred during the move. In case of a server failure, available hardware would be brought in from Waltham and the hard drives swapped out. If the operation system failed, the team would back up the software from tapes stored in the cage.
At 8 p.m., the power on the second-to-last server was tested, but soon the team realized they’d forgotten to plug in the KVM cables – fortunately, an easy fix. Minutes later, the last e-mail server was booted up. The team was tired, but satisfied that it was making good progress.
Soon a tracer showed packets were being dropped on a firewall port, meaning one server wasn’t properly translating with the firewall, so Davies checked to make sure the firewall was set up properly. Another problem was quickly fixed as the team figured out that one of the new KVM switches had a nonworking port; the connection was switched to another, and the bad port was tagged.
By 8:45 p.m., all the servers had been tested, the hardware was operating, the IP connections were good, and the operating system was up and running. They gathered around to watch Czerwinski do his magic: The plan was to bring the Exchange server databases up, and once that happened, they would know that e-mail could be delivered. Avoiding potholes on the drive over wasn’t nearly as stressful as waiting to see if all the databases came up, Lincoln says.
The staff breathed a sigh of relief at 9:18 p.m., when Czerwinski patted the last server affectionately – the one that always acted up when he was on vacation – and announced that all the databases were functional. Next came a dinner break, but the day was far from over.
The foursome continued to load back-up clients and software on the servers until the wee hours of the morning.
By 2 a.m., all of the e-mail servers were up and 2G bytes of data began flowing to individual mailboxes at Aquent sites around the world.
