Our New Data Center is A-L-I-V-E !!!!

About a year and a half ago, I blogged about how nothing compares to actually building a network. That rule certainly held true this week as we began implementing our new data center. Our work in the data center centered around the "War Room"; a conference room I setup in the adjacent building with our implementation plan on the wall, a continuous conference bridge, and morning and evening all-hands status meetings at 8:30 AM and 4 PM respectively. At each meeting, which only took 10 minutes, we would check the status of each installation area (Management Network, Aggregation Switches, Service POD, etc.) and list issues we ran into. These meetings provided focus and direction for the engineers. We also had lunch brought in each day (people gotta eat!). Despite the exhausting work of our DC Operations team, Monday started without full power in the data center yet, so we spent the morning finalizing configurations and helping to rack-and-stack equipment. By the afternoon, we were putting configurations on some devices, but focusing on finishing the rack-and-stack and cabling. By Tuesday morning we had full power in all rows and focused on the configuration of the management network so engineers did not have to "console-in" to devices inside the data center. Management firewalls and Cyclades were finalized. We also began the initial configurations of the Nexus 7000s (Aggregation Routers) and Services POD (ACE blades). The new ASR 1000 WAN routers were also racked and cabled. By Wednesday we were at full speed. The Aggregation Routers (Nexus 7000s) and Services POD (ACE blades) were configured and we started the micro Pods (Nexus 5000s). All rack-and-stack of other equipment was completed and the management network configuration was finalized. Having the management network up and being able to reach all devices via console provided a much easier work environment for the engineers. This was a key milestone for the week. Since the management network is completely separate from the data center network, engineers could log in remotely and continue working once they traveled home. We did run into some issues Wednesday. The SAN configuration had to be pushed to next week because of other, non-new, data center projects that came up. And we seem to be missing some SMF SFPs. Finally, we had to deal with the power going down each night as the building engineers continued to make final adjustments. That lead to screams of "wr mem, wr mem" around 5 PM. On Thursday we made sure software levels were correct on all devices, including doing ISSU on the 7000s. WAPs and building access switches (for PCs and IP Phones) went in even though the cabling vendor struggled with that part of the cabling. :-( The ASR 1000 WAN routers were configured and joined into OSPF Area 0. Finally, that evening, we made changes to our production Internet module at the site to support the new Extranet and DMZ inside the new DMZ. All changes went well. By Friday people were worn out. Attendance at the 8:30 AM war room status meeting was low. At this point almost the entire data center was configured and working as expected. The final step was to enable the 10GbE links from the IT Core to the Aggregation Routers (Nexus 7000s). Those links came up, and after a little OSPF stuck-in-EXSTART flakiness, started working fine. What a week it was! Just some statistics of what was done in the last team week by this amazing team:

  • Terminated and tested 148 fiber connections
  • Terminated and tested 157 copper connections
  • Installed 150 (4U) wire managers
  • Installed 150 (2U) wire managers
  • Installed 300 cabinet blanking panels
  • Labeled 150 server racks (300 doors front and back) with proper rack identification
  • Labeled 720 PDU’s with proper rack and electrical bus identification
  • Labeled 720 Starline Taps with proper electrical bus identification mapped to rack location
  • Connected 150 power cords
  • Installed 85 pieces of hardware
  • Configured two Nexus 7000s
  • Configured six Nexus 5000s
  • Configured four Nexus 2000s
  • Configured two Cisco 6500s with ACE blades
  • Configured two ASR 1000 routers
  • Configured twelve 3750 management switches and two management firewalls
  • Configured a new Extranet with four switches and new firewalls
  • Installed new 10 GbE cards in IT Core Routers
  • Consumed 124 can of soda
  • Devoured 24 pizzas
  • Swallowed 10 lbs of lasagna
  • Finished it all off with beer and wings at Hooters!

Thanks guys!!!









More >From the Field blog entries:

Phase Two of our New Data Center Project has Begun

Cisco Advanced Services Education and the Awesome Technical Knowledge Library

My Interview with Cisco about the CCDE

The Cisco QuantumFlow Processor - The Engine in the ASR 1000 Series

NPA Awards for Professionalism - Nominations Have Begun

NX-OS Training Videos on YouTube

  Go to Cisco Subnet for more Cisco news, blogs, discussion forums, security alerts, book giveaways, and more.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Related:

Copyright © 2009 IDG Communications, Inc.

SD-WAN buyers guide: Key questions to ask vendors (and yourself)