Cisco Vice President of IT John Manville discusses the company’s data center makeover strategy, the migration to the company’s own Unified Computing System, the benefits of cloud and the new IT organizational structure with Network World Editor in Chief John Dix. (Also see Cisco bets big on new Texas data center.)
Cisco’s top 10 rivals
Cisco is making some big investments in data centers. What’s the background?
About three years ago we decided we needed to treat our data centers in a more strategic way, so we put together a cross-functional team and they developed, over about a six-month period, this 200-page strategy. It was taken up to the investment committee of Cisco’s board because there was a request for a fairly substantial amount of investment in data centers to make sure we had sufficient capacity, resiliency, and could transform ourselves to make sure we could help Cisco grow and make our customers successful.
Was UCS on the drawing boards by that point?
We knew about it. It wasn’t public and you had to sign an internal NDA, but we didn’t know many details so the strategy is relatively high level and was built flexibly enough so we could take advantage of the technology without dramatically changing the overall strategy.
Does the document address timing?
It’s hard to forecast for 10 years, but because of the amount of money involved we said this is between a five- to 10-year strategy.
And you’re three years into it now?
It took us six to nine months to get all the agreement required, but we’re between two to two-and-a-half years into it. Part of the strategy was to build data centers or partner with companies that have data centers and we bundled the investment decisions in phases and got agreement for Bundle 1.
Bundle 1 mostly involved building the new data center in the Dallas area (see story) and also addressing needs in North Carolina and in California. And then Bundle 2, which we’re in the middle of, involves building a data center and getting a partner site in Amsterdam so we can have an Active/Active capability there as well.
There is a Bundle 3 for something in the Asia Pacific region if the business requirements and latency requirements require that we have something there.
Active/Active being the ability to have two data centers simultaneously support critical apps for resiliency sake, right? That was spelled out in the strategy?
Yes. The two data centers in Texas [one that had just been overhauled and a new one that is just being completed now] are configured as Metro Virtual Data Centers (MVDC). So they’re basically joined together. The strategy laid out that foundation, but that was really the second layer. The first layer is around capacity. We have to get the right capacity and then use that capacity in the right way, make sure it’s fully optimized. The resiliency piece – the Active/Active – said that with some of our applications we have to increase the overall resiliency because customers expect that and we need to be able to accommodate growth.
What kind of applications will be run in Active/Active mode?
Aspects of Cisco.com and several aspects of our ERP environment. There are other ways we provide resiliency for those at the moment, but we think going to an Active/Active configuration is probably the best way to provide the best availability. For Cisco.com, which is customer facing, it has to be instantaneous. We can’t stop taking orders.
What role will UCS play in your data center plans?
We’ve gone down the UCS path in a big way. We have almost 2,500 blades deployed globally now and have more than half of our existing virtual machines running on UCS. The actual total figure is more because we’re bringing new ones on as well as doing the migration. So we probably have over 3,000 virtual machines running on UCS now.
And those VMs represent what percentage of your total?
Cisco has, including the engineering side, approximately 16,000 to 17,000 operating system instances. So, it’s a combination of bare metal servers, physical servers and virtual machines. In 12 to 18 months, we expect to get 80% of IT managed operating system instances virtualized and have more than 90% of the workload serviced by UCS.
What platforms are you migrating from?
Basically HP and Sun. The goal is to get off HP as quickly as possible. [We had thought we would need to keep some of the] HP Superdome environments — which are like mini-mainframes — around for the foreseeable future. But with the new Intel chips that are coming to UCS, in the lab we have shown we can move even that environment to UCS. And while we haven’t done that yet — we’re still going through a couple of proof of concepts — we’re reasonably confident we can handle our biggest transaction volume databases on UCS clusters. The Sun [gear is] totally in development but it looks like we can move to UCS in that space as well.
So, I think this is a pretty good story for us. That a $40 billion company like Cisco can effectively run on top of UCS.
Are you essentially using UCS to build a private cloud?
Right. Our internal cloud is called CITEIS, or Cisco IT Elastic Infrastructure Services, and it’s basically Infrastructure as a Service, combining compute, storage and networking. CITEIS should be able to service 80% of our x86 requirements. We think there’s still going to be some point things — some real high-end production databases — that we’re going to have to serve with dedicated environments, and maybe not even virtualized, so using UCS as a bare metal platform.
We also think we’ll be able to work with an external service provider and have some form of either overflow or move capability so our business users won’t know if their compute and other capabilities are being delivered internally or externally. That’s a stretch to provide that because you have to make sure the latency is OK and the security requirements are OK, but we feel we have to start to investigate that and hopefully do it within the next 12 months.
On the virtual software side you have mostly all VMware stuff. What’s your thinking about the need to support Microsoft and other virtualization tools?
The virtualization piece of CITEIS is based on VMware. We’ll be open to other hypervisors, but at the moment we haven’t gone down that path. If there were additional capabilities one of the other ones would give us we’d be open to doing that.
Regarding Microsoft, what operating system will the bulk of these UCS deployments be running?
It depends on what the application needs. It’s approximately 50%/50% between Linux and Windows. There’s no religion about using Windows or using Linux.
What benefit do you expect to derive from the cloud?
There’s a lot of hype, [but the basic question is], is there anything over and above the benefits provided by virtualization? We tried to work out the TCO, and these TCOs are using discounts an enterprise the size of Cisco could expect, so we think the figures are customer repeatable.
The TCO here includes everything that we could think of that goes into providing the internal cloud except for the disk storage. So the SAN capabilities and the MDS capabilities are included, but the actual disk is not because we felt each application uses a variable amount of storage and the amount and cost is very application dependent. So, we left that out, but we included everything else … operations, space, power, people, etc. And what we’ve managed to show is that the cost of having a physical server inside the data center was about $3,600 or $3,700 per quarter.
When we went to virtualization, the average TCO — because not all systems could be virtualized — went down by a pretty good percentage, about 37%.
We think once we implement UCS and the cloud technology we can get that down to around about maybe $1,600 — on average — per operating system instance per quarter. Where we are right now is somewhere in the middle because we’re still moving into the new data center. And we still have a lot of legacy data centers that we haven’t yet totally retrofitted with UCS or our cloud. But even so far, we’ve managed to get the average TCO down by another 15%.
So we think there is a benefit from cloud and UCS. If you ask me to say approximately how much is from UCS and how much is from cloud, we did that analysis and it’s about half and half. So, half of the savings going from $2,300 down to $1,600 is due to UCS, and about half is from the cloud type constructs, which are automated provisioning, some automated management, in some cases self-service provisioning, things like that.
If we get a little bit more aggressive about virtualization and squeezing applications down a little bit more, we think we can get the TCO down to about $1,200 per operating system instance per quarter.
The other benefit is agility. We get much, much quicker provisioning. The goal is to get to 15 minute self-service provisioning. And we have this running but haven’t turned it over to the application developers for various charging and other authorization issues. But our sysadmins are using this and they’re seeing significant productivity gains by being able to provision virtual machines in automated fashion.
Once you have built out all these shiny new data centers, can we expect to see Cisco offer any public Infrastructure as a Service offerings?
Obviously John [Chambers] and the other executives could change this, but it’s not Cisco’s business strategy at the moment to offer Infrastructure as a Service. What we’ll do is provide the capabilities for service providers so they can offer that.
As if you didn’t have enough on your plate with all the data center work, I understand you also remade the IT organization as well. What happened there?
We changed from a silo organizational structure in the base infrastructure teams, which is what I’m responsible for, so we don’t have a network team and a storage team and a platform team anymore. We have an architecture team, a design team, an implementation team and, effectively, an operations team.
What we were seeing before was sub-optimal system level designs. So, we had networking optimally designed for just the network, and the platform group designed optimally for platforms. But as an end-to-end system, it wasn’t optimally designed and we decided to change that.
There were a few teething pains but I really feel that, while it’s not perfect, this is a much better organizational structure, especially as you get closer to virtualization.
Because we are aggressively moving towards becoming a services-based organization, we are overlaying on top of this virtual service teams. So there’s a network service and we have a service owner for that, which in this case happens to be someone on the design team, and that person has virtual members in all the other groups and their role is to make sure the overall network service meets the requirements.
It took about a year to make the change and I can tell you, it wasn’t easy because various teams were worried their role was going to get taken over by other people. In the end, I think most people have realized their role becomes richer and they can have more direct impact on running this organization.
What defines what computing people go into architecture vs. design, vs. implementation? How do you sort that out?
It depends on skill set and also what the motivation of the person is. Some people don’t like to rack and stack and configure things, which is what the implementation people do. Some people like thinking two or so years out, which is what the architecture team generally does. So where people go is partially driven by their skill set and partially driven by what they want to do.
There are also the realities of not being able to have everybody in architecture and nobody in implementation. So, of course, we had some influence over who went where.
