U.S. data centers rack up billions in electricity costs annually. How to get power costs under control before the energy crisis hits you
Adam Gray, CTO of Novacoast, isn't responsible for the power and cooling costs of his company's two data centers, and he isn't too worried about the effect of escalating energy prices on his IT operations. Novacoast is a Santa Barbara, Calif., IT professional services firm with 100 employees and 17 locations.
"California, where our primary data center is located, has pretty well stabilized power costs in the last few years, so we haven't seen any big increases," Gray says. "We have an infrastructure in place to grow without having to add a lot of servers. [Power costs] are not a major driver for us."
One reason Gray isn't too worried about power costs is that Novacoast recently finished a server virtualization effort, migrating from 25 1U servers to two blade servers. Now Novacoast can turn up virtual servers in minutes to support software development efforts for its clients. Virtualization also has reduced the firm's monthly electric bill by a few hundred dollars.
Gray says he'd pay more attention to energy use if his electric bill was higher. That's the reason he hasn't factored energy efficiency into his ongoing evaluation of servers from Dell, HP and IBM.
"Energy efficiency is not a priority," Gray admits. "Our priorities are support, hardware replacement, mean time between failures, cost and reliability. The power consumption cost is a very small percentage of our IT costs. But if we lost a server, that would be catastrophic. Energy efficiency is important, but it doesn't make our top five."
Gray is not alone. Most IT executives haven't focused on their IT equipment's power costs, and they aren't taking energy efficiency into account when they choose servers, storage devices or network gear.
In a recent survey of 590 Network World readers, 68% of respondents said they were not responsible for power bills related to their data center's IT equipment, and only 21% had established an ongoing dialogue between IT staff and facilities management personnel.
A majority of IT executives - 51% - don't consider energy efficiency in IT product evaluations, the survey found. In addition, more than 50% of the respondents failed to take the most obvious steps to reduce IT power consumption, such as removing servers no longer in use. The e-mail survey was conducted in November 2007.
This lack of interest in IT power costs appears to be changing, particularly at Internet companies, financial institutions and leading-edge retailers.
More IT executives are coming to grips with a grim reality: Data-center power and cooling costs are the hidden enemy of IT departments. They creep up on unsuspecting CIOs like deadly mists and choke off their ability to deploy new equipment and applications.
"If a CIO has not had to build a new data center recently, this is likely to be a huge surprise," says Ken Brill, founder and executive director of the Uptime Institute, which provides consulting services to more than 100 data center operators.
"Oftentimes, the people who pay the power bill aren't in the IT department, they're in the facilities department. Where it shows up is in the capital cost for the data center," Brill says. "This all happens invisibly until you run out of capacity."
CIOs who get data-center power and cooling under control can reduce IT operations costs significantly, bolster corporate profits and gain a strategic advantage over their competitors, experts say.
"There's a huge opportunity here. By looking at efficiency and improving your operations, you can substantially reduce the cost of your data center," says Christian Belady, principal power and cooling architect for Microsoft's global foundation services. "As IT operations become a bigger piece of the cost pie, how well businesses manage their operations and how efficiently they run their data centers could be the difference between making money and not making money, between having a lower cost structure than competitors and beating them on Wall Street or not," he adds.
The data-center power problem
The cost of a data center's power and cooling typically is more than the cost of the IT equipment inside it, experts say. That's because today's IT systems - including servers, routers and network-attached storage - pack more transistors on each chip and more power-hungry chips in the same or smaller footprint.
"When you buy computer equipment, it comes with an embedded level of power consumption," the Uptime Institute's Brill explains. "Let's say you spend $10 million a year on hardware. The same $10 million that you spend today will bring with it 10 to 15 times the power consumption it did in 2000."
Meanwhile, companies are demanding more compute cycles, and that forces them to add servers. Although today's servers are more energy efficient than earlier models, improvements in energy efficiency haven't kept pace with increases in computational performance, experts say.
Today's IT hardware requires more UPS, generator, air conditioning and power-distribution capacity than in the past. That's why IT executives looking to deploy servers often are surprised to find out that they have run out of space, power or cooling capacity in their data centers.
"The average CIO is only marginally aware of his electricity bill but is keenly aware of the powering and cooling limitations in his data center facility," says Carl Cottuli, vice president of American Power Conversion's (APC) Data Center Science Center. "Over the last couple years, this issue has really come up to the front seat. It's always been there, but it was less of a concern than other issues on an IT manager's desk. Now it has a direct impact on the ability to deploy enterprise-class servers, and it's driving virtualization and consolidation efforts."
CIOs who aren't managing their data center power and cooling capacity can get caught off-guard. "I have seen IT managers who weren't able to roll out new equipment [they have ready to plug in] because they had no available power or cooling," Cottuli says.
Part of the problem is that most data center operators aren't measuring or tracking the energy efficiency of their buildings. The Green Grid, an IT industry consortium, has come up with two new metrics - Power Usage Effectiveness (PUE) and Data Center Infrastructure Efficiency (DCiE) - to measure how well a data center manages the power and cooling overhead required by its IT equipment. At this point, however, only a handful of data center operators use these metrics, experts say (see "Two ways to measure power consumption").
"You can't improve it if you don't measure it," Microsoft's Belady says. "For years, I went around talking with various customers about best practices. When I visited them a year later, they hadn't changed a damn thing because they couldn't quantify the benefits."
In 2007, however, companies started getting pressure from boards of directors and shareholders to reduce energy use and become more "green." As soon as companies take a look at their electricity bills, they figure out that their data centers are their heaviest users.
"The data center consumes up to 40 times more power per square foot than anything else in the asset portfolio. When you do an energy audit, this is going to jump out," the Uptime Institute's Brill says. "For a large company, hundreds of millions of dollars could be saved over 10 years by taking steps to make data centers more energy efficient." (See "Five ways to power down in the data center.")
The data center power problem is serious enough to attract attention from policymakers. Between 2000 and 2006, the amount of electricity consumed by U.S. data centers doubled, and it is projected to double again by 2011, according to a U.S. Environmental Protection Agency report to Congress last summer on server and data center efficiency. The EPA estimates that data centers consumed about 61 billion kilowatt-hours of electricity in 2006, which cost $4.5 billion and represents 1.5% of the total bill for electricity used in the entire United States.
IT-related energy use is becoming a higher priority for CIOs for two reasons: rising electricity costs and pressure to reduce the IT carbon footprint, says Jack Pouchet, director of energy initiatives at Liebert, a leading provider of data center cooling systems.
"If the cost of electricity went down to 3 cents a kilowatt-hour, the cost concern goes away. Then it's a question of best practices. People don't want to waste money . . . and build a data center that's twice as big as they need because electricity is cheap," Pouchet says. "But CO2 - the green issue - that is on everybody's radar. That issue doesn't appear to be going away."
Testing for energy efficiency
Attacking the data center power problem starts with buying energy-efficient IT hardware, power and cooling equipment.
"There is energy-efficient hardware out there, but it costs more," the Uptime Institute's Brill says. "All the manufacturers tell us that their energy-efficient hardware isn't selling."
State Farm Life Insurance in Bloomington, Ill., is one of the few companies to take energy efficiency into account when it purchases servers, PCs and other network gear. The company's facilities team has set up a laboratory, run in conjunction with the IT department, for testing power consumption of all the systems going into its data centers. It prefers testing the equipment itself to relying on vendor-provided energy-efficiency statistics, which have proved inaccurate, says Ron Kalley, the company's director of facilities. "The only way for us to figure out which server or PC is more energy efficient is to put it in our own shop on our network, and to test that component in the way we're going to use it to see what it does for us," he says.
Last year, for example, State Farm tested power supplies as part of its procurement of 200,000 workstations. The company found that if it spent $15 more per workstation on a more efficient power supply, it would earn back that investment in the first year with reduced energy costs.
"We've gotten folks to [understand] that if we do our due diligence and look at the power consumption of equipment, we can save money overall," Kalley says. "Our workstations are on a three-year rotation, so for years two and three, the power is basically at no charge."
State Farm has tracked its IT operations expenses since it unexpectedly ran out of capacity in one of its data centers in 2004. The company runs data centers in Bloomington, Atlanta, Dallas and Phoenix.
"In 2004, we started recognizing the associated costs of the IT capital spend," Kalley explains. "This was really a catalyst for us to start thinking more clearly about what we needed to do to manage [capital expenditures and operating expenditures]. We needed to get more proactive at looking at life-cycle costs."
State Farm's initial focus was on better managing data-center capacity to make sure it wasn't caught flat-footed again. Now the company plans its data-center capacity at least 24 months ahead of time so the facilities team can meet the needs of the IT department.
"Around 2005, 1U, 2U and 3U servers were becoming commodities, and people were buying them like M&Ms," Kalley says. "IT folks were buying as many as they could at the lowest cost, but they weren't looking at the power supplies. We had to find a way to stem this tide."
In its laboratory, the facilities team tests not only the power consumption of equipment headed for its data centers but also its performance and reliability. Among the equipment tested in the lab are PCs, servers, power distribution units, static transfer switches, network equipment and storage devices. Still, measuring energy savings from the test lab is hard, Kalley says. "All the time we're trying to drive kilowatt consumption down, the price of energy is going up. I think the benefit is in cost avoidance rather than cost savings."
One advantage for State Farm in attacking data center power consumption is the close working relationship between facilities and the IT department. The facilities group shares the data center electric bill with the IT group, and the two partner on testing, procurement and other activities. As director of facilities, Kalley spends half his time working with the IT department.
"It's not just energy consumption. The other thing we're trying to do [with the IT department] is socialize the understanding that capacity is not infinite," Kalley says. "We've been trying to manage our consumption to the best of our ability so we can [wait] four, five or six months before we have to spend multimillion dollars on an upgrade to one of our facilities. That's the cost-avoidance piece."
Companies that manage data center capacity well will end up meeting corporate carbon-footprint requirements, too, Kalley says. "At State Farm, the energy spend for our data centers is 25% of our entire real estate portfolio," he says. "Everybody is talking about going green. I'm just talking about hard-nosed, efficient operations. If you do that, you'll be green."
Focus on metrics
Data-center operators must start measuring and monitoring data center power use in real time, experts recommend.
One key measurement is The Green Grid's PUE, which shows the ratio of the power used by a data center's IT equipment to the power used by its power and cooling systems. (DCiE is the reciprocal of PUE; it shows the amount of power going to IT equipment as a percentage of the total power going into the building.)
Data center operators should aim for a PUE of less than 2 and ideally as close to 1 as possible, experts say. In other words, if you need 1,000 watts of power for your IT equipment, your data center should require no more than 2,000W overall.
Wachovia, the nation's fourth-largest financial institution, has a high-end data center in Birmingham, Ala., that operates at a PUE of 1.6. "We're in the stage of designing new data centers, and we're trying to get the PUE down to 1.4," says Bob Cashner, senior vice president of corporate real estate for Wachovia, in Charlotte, N.C. "Ideally, we would get that number down to 1. We're aiming for that. We're trying to do things that get our PUE lower and lower."
For the Birmingham facility, Wachovia used the most energy-efficient UPS systems, generators and chillers available at the time, Cashner says. "We look at the lowest long-term owning and operating costs. You can spend a few more dollars on Day 1 and do things that will save you money long term."
Wachovia is using virtualization in the data center to reduce the number of servers it needs, replacing 16 individual servers with one virtualized server.
For its next data center, Wachovia is considering using DC power for its IT equipment rather than the traditional AC. This would eliminate the need to deal with the "transformation losses of the UPS systems," Cashner says.
One reason Wachovia is a leader in data center efficiency is that the facilities and IT departments work closely together. "One of the things we've done for years is to get around the table all the subject-matter experts - risk, security, technology and corporate real estate - so we can come up with the best solution that balances all the different factors," Cashner says.
This relationship will be important as data center costs continue to rise. For example, Wachovia's 225,000-square-foot Birmingham building, which opened in 2006, cost $112 million. If it were to be built today, the same building would cost $182 million, Cashner says.
"One of the things Wachovia has done really well is capacity planning," Cashner says. "We have a better idea of what the load growth is in our mission-critical data centers by having a good pipeline of what's coming down the pike in the business units. This lets you have new data centers come online just in time. It takes a good working relationship among the different stakeholders."
Interest in data energy-efficiency is at the "highest levels" within Wachovia, Cashner says. "We have a commitment from our CEO on down that we are going to be a green organization. We have a laser-beam focus on energy efficiency, and we've had that for a long time."
The power risk for CIOs
Companies that don't measure and improve data center efficiency as Wachovia and State Farm have done risk losing their competitive edge, experts say.
Companies that don't improve data center efficiency are "going to go out of business because the cost per transaction will be prohibitively expensive for them compared to the competition," Liebert's Pouchet warns.
Noting that the U.S. Green Building Council soon will be rating data centers, Pouchet predicts companies will be out of favor unless their data centers are rated silver, gold or platinum. "People will start using data center ratings as a metric to get your business. In one to three years, this is going to be something on people's business cards: We run a gold-certified data center."
The data center power and cooling problem is going to get worse before it gets better, experts say.
The Uptime Institute's Brill estimates that the cost of providing power and cooling to data center equipment is now one and a half times the cost of the equipment itself over its lifetime. "I see it growing to three times or four times," he says. "The problem is that the growth is invisible until the data center runs out of capacity."
CIOs who think they have solved the data center power and cooling problem through server virtualization are wrong, experts say. Virtualization is a one-time fix. It can help you delay dealing with data center efficiency, but it won't fix the problem forever.
"Virtualization will get energy costs below the threshold for a while, but it will pop up again," Microsoft's Belady says. "Virtualization buys you time, but after that virtualization won't give you more energy efficiency."
Data center operators who can figure out how to eke out more efficiency from their facilities continually are going to be an asset to their employers, experts say.
"The efficiency methodologies enable you to get more compute capability out of the same kilowatts," APC's Cottuli says. "A business manager may need 5,000 transactions per second. If the data center manager can put in some efficiencies, such as better cooling or virtualization, and give the business manager the extra transactions with the same amount of kilowatts, that's a win-win."
The data center power and cooling problem "fundamentally changes the underlying economics of IT," Brill says. "CIOs who don't adapt to this new math could make profound investment mistakes."