Why your new data center doesn’t work; the sorry state of commissioning

Data center capital projects go sideways more often than not, and executives diminish the only process that could prevent it

Why your new data center doesn’t work; the sorry state of commissioning

For enterprise IT organizations embarking on a data center capital project, the stakes are undeniably high. Building a new data center is a massive investment, but it also enables or hampers an organization’s IT strategy and capability—affecting an organization’s business performance for years to come.

As more organizations rely on colocation data center providers, ensuring the design and construction of these projects meet your business requirements is critical as well.

+ Also on Network World: Envisioning a 65-story data center +

With multiple vendors, subcontractors and typically more than 50 different disciplines involved in any data center project—structural, electrical, HVAC, plumbing, fuel pumps, networking and more—it would be remarkable if there were no errors introduced or corners cut during the construction process.

Lapses in construction oversight, planning and budget can mean an expensive new data center facility will fail to meet the owner’s requirements, with the end result offering poor performance or limited flexibility.

Addressing problems may delay the project and usually requires significant spending. In some cases, the problems continue to hamper operations for the life of the data center and may eventually require the facility to be replaced prematurely.

Even if the facility should continue operations for its expected life, it may cost more than expected to operate, suffer more downtime incidents and complicate efforts for the owner to introduce new products and services.

With so much riding on an IT infrastructure project, and given the maturity of the data center industry, you might assume that capital project errors are rare.

You would be wrong.

Data center projects fail constantly

Organizations such as Uptime Institute evaluate data center capital projects to ensure the data center’s business requirements and goals are met throughout the data center design, construction and turnover to operations.

Typically, technical staff go to the new data center site immediately after a phase called commissioning—the heart of most IT capital project failures.

Non-technical project owners may view commissioning as an arcane, overly engineered process that they can diminish or shorten to meet other business pressures. But they do so at their peril, as commissioning is the most important phase in determining the success of a project.

Commissioning:

  • Verifies that the equipment and systems operate as designed by the engineer-of-record
  • Provides a baseline for how the facility should perform throughout the rest of its life
  • Affords the best opportunity for operations to become familiar with how systems operate and test and verify operational procedures without risking critical IT loads.

The site is supposed to be ready to go live after commissioning.

Yet, every exit report from the last 200 Tier Certifications reveal some design flaw, equipment failure or unexpected problem with the data center infrastructure uncovered during the Tier Certification process. In fact, more than 70 percent of projects fail Tier Certification assessments during the first round of demonstrations, requiring the owner to invest more time and resources to remediate the problem.

During Tier Certification on a capital project in the Americas in 2016, backup power systems failed during a simulated electrical utility outage. This was an anticipated design condition—and arguably the most rudimentary function—of the new data center. The underlying reason for the failure was a “feature” engineered into the backup power systems, but the owner did not receive training, did not have appropriate knowledge, and had not been informed of its existence, thereby defeating the purpose of the data center.

During a 2016 Tier Certification of a project in Europe, Uptime Institute discovered the data center posed a threat to life safety. Service work on the power systems necessitated placing a screwdriver on a live 400-volt connection. Additional failures were discovered, attributed to incorrect fuse ratings and errors in the building monitoring and automation system. Any of those three issues would have resulted in a service interruption of the new data center.

“Many data center owners rely on Tier Certification as a commissioning event,” explained Chris Brown, Uptime Institute’s senior vice president of tier standards.  “Owners conduct minimal testing before we arrive and are counting on us to commission the site, which the Tier Certification is not intended to.”

It is important to note that Uptime Institute’s assessment was the very last step in the capital project, immediately preceding the new data center being placed into service. All of the capital project stakeholders had signed off on the data center before the assessment began.

So, why weren’t these issues discovered during the commissioning phase?

The sorry state of commissioning

Commissioning activities represent a unique opportunity for data center owners to rigorously test the capabilities of the critical infrastructure. But far too often the process is shortchanged.

Uptime Institute consultants place the responsibility for commissioning failures on the owners and project management team, rather than the commissioning agents.

“Data center owners put a leash on their commissioning agents to keep cost down and maintain schedule,” said Ryan Orr, Uptime Institute senior consultant.

Delaying the opening of new data center whether enterprise, colocation or wholesale is not an option for most companies. So, if commissioning is not completed by the handover date, IT starts moving in anyway. When IT starts moving in, they run the place and can, in many instances, make commissioning activities difficult to complete properly.

“Commissioning is always the last thing that gets done,” Orr said. “So, if there are project delays, which every project has, commissioning is always the squeeze point. Or if the project is going over budget, commissioning is the squeeze point. What started as a two-week schedule gets squeezed to three or four days, and the commissioning agents have to do the best with what they have.”

Orr pointed to several other problems in the current state of commissioning:

  • Too many organizations are comfortable with assumptions and representative testing. For example, an owner may commission the first phase of a project, but in follow-on phases cut commissioning short because of the shared systems and owner’s do not want to put live IT loads at risk testing shared infrastructure.
  • Inappropriate relationships between contractors and commissioning agents continue. Owners should hire commissioning agents directly. However, that seems to happen less and less and raises severe concerns about conflicts of interest.
  • Many owners believe they do not need to commission rigorously because they can rely on infrastructure redundancy. Yet sometimes that redundancy is fact and sometimes its fiction—and the owners will not know unless it is tested properly. 

How to do commissioning right

A well-planned and executed commissioning program will help validate the capital investment in the facility to date. It will also put the operations team in a far better position to manage and operate the critical infrastructure for the rest of the data center’s useful life—and ultimately ensure that the facility realizes its full potential.

Construction teams that are insufficiently experienced in the rigors of data center commissioning often underestimate the time required or regard the commissioning period as a kind of buffer that can be accessed when work runs late. For both of those reasons, it is important that the owner or owner’s representative take care to schedule adequate time for commissioning and ensure that contractors meet construction deadlines. A recommendation would be to engage the commissioning agent and general contractor early in the process as a partner when developing the project schedule.

In addition, data center capital projects include requirements that might be unfamiliar to teams lacking experience in mission-critical environments; these requirements often have budgetary impacts.

For example, owners and owner’s representatives must scrutinize construction bids to ensure they include funding and time for:

  • Factory witness tests of critical equipment
  • Extended Level 4 and Level 5 commissioning with vendor support
  • Load banks to simulate full IT load within the critical environment
  • Diesel fuel to test and verify engine-generator systems

Because experienced teams understand the importance of data center-specific commissioning, the commissioning agent will be able to work more effectively early in the process, setting the stage for the transition to operations.

In addition, operations should be part of the design and construction team from the start of the project through commissioning and handover. Including operations in change management gives it the opportunity to share and learn key information about how that data center will run, including set points, equipment rotation, change management, training and spare inventory.

Data center capital projects are subject to complex challenges, with multiple stakeholders and contractors coming together across multiple disciplines. To ensure the infrastructure investment meets an organization’s business requirements, project leaders need to select the right partners, empower a competent owner’s representative and leave adequate time for rigorous commissioning and third-party certification.

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.
Must read: Hidden Cause of Slow Internet and how to fix it
Notice to our Readers
We're now using social media to take your comments and feedback. Learn more about this here.