Data center moving day

With a 24/7 e-commerce storefront and more than three dozen stores spread across the mid-Atlantic region, Boscov’s doesn’t have time to close for a data center move. So, when technical support manager Joe Poole heard about an impending migration, he knew the project would be daunting. Careful planning was the key.

With a 24/7 e-commerce storefront and more than three dozen stores spread across the mid-Atlantic region, Boscov’s doesn’t have time to close its doors for a data center move. So, when technical support manager Joe Poole heard rumblings about an impending migration, he knew the project would be daunting.

For one thing, the atmosphere around the move was different from what Poole had experienced when Boscov’s shifted data centers in 1989. “I don’t think they could have spelled ‘Web’ back then,” Poole says of the retailer in North Reading, Pa. “I think we just shut things down and did the move on a Sunday.”

It wouldn’t be so easy this time. With Boscov’s operating as an always-on business, the challenge was to transfer the company’s IT operations without causing a major outage or lengthy downtime. There were terabytes of data residing on disks and a tape library that had to be moved, a project Poole estimated would take at least 24 hours.

“Without the tape library there was no batch production, and there was never going to be a day – especially as we were looking at September, when we start to get busy for the Christmas season – that we could be down for 24 hours,” he says.

Boscov’s was moving its IT operations to become the inaugural tenant of a new IT service provider, Directlink Technologies. Directlink is one of several projects launched by Boscov’s owner, Albert Boscov, supported by Gov. Edward Rendell of Pennsylvania and aimed at revitalizing an area hit hard by job losses.

Housed in the former General Public Utilities building in Reading, Directlink has more than 80,000 square feet of data-center space, as well as access to a broadband fiber-optic network that reaches into the Northeast and Middle Atlantic and Midwestern states.

“In order to make [Directlink] viable to potential customers, Mr. Boscov . . . decided there should be a presence in there,” Poole says. “That’s where we came in.”

The new location – with raised floors, multiple power supplies and backup generators – represents a step up from what Boscov’s was operating in before. “We had been in a facility that was built in the 1920s for Rosedale knitting mills,” he says.

Although the physical distance between Boscov’s and the data center was just 10 miles, the logistics were overwhelming. With careful planning, however, Boscov’s completed the move more quickly than expected and ended up with an upgraded infrastructure that will support the rapid growth of the company, Poole says.

Poole adds that the retailer now is positioned for long-term savings, though he wouldn’t disclose specifics. “As it turned out, we’re going to open 10 stores this year, so the extra capacity was well worth it,” he says.

When rumors of the move started circulating in March and April 2005, the first thing Poole did was to call business partners Distributed Systems Services, IBM and Sirius Enterprise Systems Group. Boscov’s realized it was ready to upgrade equipment, and its leases on existing equipment were set to expire in March.

“We were lucky, in the sense that our equipment leases all ran out at the same time . . . and there were not that many months remaining on the leases that would have to be rolled over,” Poole says.

Boscov’s would bring into the new data center a new mainframe, a new direct-attached storage device (DASD) and a new tape library with upgraded drives, and keep its existing equipment running in the old location.

It was this kind of infrastructure musical chairs that let Boscov’s move without suffering an outage. To support that, Boscov’s network team had to recreate its existing network in the new facility and then link the two.

“We had to build a whole new infrastructure of Cisco routers. This had to be a seamless operation, meaning that our network that was in [the old data center] had to reappear in the new facility like nothing ever happened,” Poole says. “You couldn’t go in and redefine everything. You had to move it. So our very talented networking crew was able to bridge two networks together 10 miles apart so people on either side of the fence would be able to access the system.”

Boscov’s settled on Sept. 24 as moving day. At 7 a.m. Poole and his team began moving z/VM data from mainframe storage to the new IBM DS8100 DASD Shark at Directlink. The Web site was offline for this move but was back up by 11 a.m.

A similar process was used to move z/OS data to the Shark, though the Web site continued running through this move, which was completed about 4 p.m. The old z900 continued running the show, though it was accessing data at Directlink remotely via fiber.

Once the stores closed at 9:30 p.m., the z990 was brought online, and the z900 was retired. The tricky part was to make sure the machines didn’t confuse things because they looked the same to the network. Boscov’s network specialists handled this with VPNs and routers.

“That’s one of the things you really have to think about when doing something like this: not messing up your network,” Poole says. “Make sure that is thought through, because you’re going to have pieces of equipment with the same names that may be running at the same time. Think about how you’re going to filter out traffic that shouldn’t be going into your production world.”

IBM had already assembled the tape library at Directlink, so once the data was moved and the new mainframe turned on, the hard part was done, Poole says. The equipment in the old data center was disconnected and packed up for shipment back to IBM.

As for staff, servers and other peripherals such as printers, the transition was more gradual. Over the next few weeks, the staff packed up the contents of their existing cubicles and transferred them to cubicles in the Directlink facility. Boscov’s 80 servers were moved a few weeks later.

“Office functions don’t normally get used over a weekend, so we waited several weeks and picked a Saturday morning after everybody had recovered,” Poole says.

In all, a project that normally would have taken three to six months was done in less than two months without a major outage, Poole says. A big reason is that Boscov’s took the time to plan the move carefully and reassure staff along the way.

“When it gets down to crunch time, people tend to get a little nervous, a little upset, and there is a great feeling they’re not going to be able to pull it off,” Poole says. “It’s a matter of reassuring everybody that this is going to work. Management has to have the positive attitude. When you know what you’re going to do and you know the pieces are in place, it will be successful.”