On the CME Globex network, a few milliseconds matter

CME Group is the world’s largest exchange, with trading volumes averaging more than 11 million contracts per day in July 2007. Formed by the recent merger of Chicago Mercantile Exchange Holdings with CBOT Holdings, the company offers a marketplace to trade products that range from agricultural goods to real estate investments.

Electronic trading at CME has seen a compound annual growth rate of 300% since 2000. How do you keep up with that kind of growth?

The way we’ve dealt with it is we’ve been on the leading edge of WAN connectivity. Where we used to use a 56K line, we’re offering our customers 20M and 40M circuits.

Who are your customers?

Big banks, big FCMs [futures commission merchants], UBS, JP Morgan, Morgan Stanley, those types.

Can you walk through the network setup? I understand you’ve got three major data centers in the Chicago area, another in London, plus six hub sites throughout Europe and another hub in Singapore, correct?

Right. Our remote facilities are all connected by multiple diversely routed DWDM rings. Each ring is equipped with multiple load-balanced 10G Ethernet links. The rings are configured to provide maximum resiliency with minimal transport latency. We’re in the microseconds when we talk about going between data centers.

How many servers do you have, and what is the breakdown in terms of the types of servers you use?

We’re at about 3,600 servers. We count one system as a server, but one system could have 16 CPUs. Of the 3,600, about 3,200 are Unix and Linux, with Linux being 80%. And the rest are Windows and Novell servers. And we have approximately 18 Tandems.

Last year, Network World wrote that trades execute in 50 to 60 milliseconds on average, down from about 140 millisec in January 2004. What kind of work have you done in optimizing performance to get those kinds of numbers?

We have new numbers now. On our options platform, we’re using a different type of engine. It’s in the range of 3 to 5 milliseconds for matching, so it’s a lot faster. On futures, we’re on the same platform we were on, but now we’re in the 20- to 35-millsecond range. So on each platform, we increased our speeds and dropped our latencies.

With the futures engine, it’s been centered mostly around optimizing the code. We always go to new hardware when a manufacturer has newer, faster hardware. But beyond that, it’s how quick we can make the code that does the [trade] matching.

Can you walk through how an order is typically processed?

There’s a multitude of ways that a user could input an order. They could do it via a keyboard, through any front-end system. The most difficult thing to keep up with is the ATSs, the automated trading systems. That’ll be someone who has written a program and is feeding the program with a quote stream. The program analyzes the quote stream and, based upon that, calculates the next order to put in. They could put orders in at, say, 60 per second. The order goes into an external gateway, which is like the demarcation point between the customer and where CME takes over. Then the order goes to another gateway that does a protocol conversion, then it goes on to a publish/subscribe bus.

We then have a multitude of our match engines sitting on those buses that are listening for their commodity code; or in the case of options the match engine listens directly. One server might listen for an XYZ S&P future and will only pull off that bus the XYZ S&P future orders. That order then gets pushed into the match engine. The engine has an order book and it sees if it has a matching side. If it matches the trade automatically, it’ll send an acknowledgment back to the user that the trade was matched. If it can’t find a match, it sends an acknowledgment back that the trade has been received and it’s working. On the back end of the engine, you have your quotes that go out. So every time a trade is matched, it results in a quote. And that quote disseminates through another environment and out to the customer.

And all of that happens in a few milliseconds?

Yes.

Do you do anything to the machines themselves to try to optimize them?

Outside of code optimization, we do everything possible for operating-system optimization and then all the peripherals. For critical parts of data that we write to often, we’re in the process of moving those blocks of data to solid-state storage arrays. [Ed. note: Solid-state storage is based on integrated circuits, requires no moving parts and is typically faster than tape or optical storage mediums.] We already do it on the options-based engine, and we’re starting to do it on the futures-based engine. The futures-based engine is based on a Tandem platform. We were one of the first customers in the world to start using SAN storage on that platform. So between SAN and the solid-state arrays, all the talking between any storage is greatly increased in speed.

What other steps are involved in optimizing, such as with the OS?

It’s a matter of distributing resources as you need them. There’s kernel-tuning parameters that are adjusted, plus things in the TCP/IP layer. It’s tweaks here and there. We do a lot of [processing] in memory. I don’t think it’s a secret that you do stuff in memory; it’s quicker than doing it out to disk, even if it is SAN and solid-state storage.

Optimization also extends to reliability and CME’s trading systems are rated at five 9s reliability, which is phone-company territory. What are some of the key steps you’ve taken to achieve that?

The key step is we have a team that reviews what we call single points of failure, and we eliminate them. When you start with Tandem, the hardware itself is redundant. In the Linux world, we’ll have multiple servers that are clustered, so you have redundancy there. And then it just works its way back. If I have a switch, and I’m talking between the order-engine environment and the match engine via one path, the backup path for that engine and for that order entry environment is going to be on a totally separate network switch. So we could in essence lose a whole network switch, and we should have failover within seconds to the backup path. That’s what we do across the board. For every server, let’s say a gateway server, there’s a backup hot gateway server. And we’re using a publish/subscribe protocol, so if we have a failure of a primary server, the secondary server automatically starts reading that data and passing it on.

What kinds of power and cooling issues are you facing in your data centers?

We built our two main data centers with power and cooling redundancy in mind, but also a high power rating per square foot. We can basically go very dense in our data centers. That’s one of the problems that people have. If they buy blade systems, they can put one on one side of the data center and the other [has to go] on the other side. We can go pretty dense per rack, both with power and with cooling.

We’re leveraging new Liebert XDO [rack-cooling] technology that does cooling from above, with a liquid vs. an outside cooling source. We were the first customers using those. We have like the first 10 serial numbers that they started to make. We stay on top of all this stuff.

As far as saving money, we have one data center equipped to process outside air. We’re able to shut off the air conditioners for most of the winter because we just take the outside air that’s already cool and process it, to control things like dust and humidity, and use it within the data center.

How old are your data centers?

The oldest data center that we’re still utilizing, although it’s not production critical, was built out in 2000. The other two data centers, one was built out in 2003, but additions to it have been going on since, and it was completed at the end of last year. Our other data center, we started building it out in 2005, and we’re still building it out. We get data-center space and then put in all of the critical infrastructure up front, as far as the things [like power and cooling] that you only want to do once. And then we add what we call pods as we need them. I don’t necessarily build out a whole data center and have 1,000 racks waiting there to be used. I’ll build a pod out, which might consist of say 40 racks, and when we get 50% through utilizing that, we’ll start building the next pod. We call it the build-as-you-go approach. There’s cost-savings involved, because you don’t have all that hardware just sitting idle, waiting to be used.

To what extent are you using virtualization to help optimize your environment?

We’re doing that more in our development environments. We have real critical requirements for our production systems. In essence, I could have a production system running at 5% CPU use, and the [government releases] the unemployment number and there’s a big change and [use] jumps to 75%. Let’s say I was running four virtual servers on that box and they all had that jump. The CPU would get flooded and run out of cycles to give out and the user would experience latency. That’s unacceptable in our world. But in our development and QA environments, it’s acceptable to do virtualization.

What kind of benefit has that given you?

We are in the middle of doing it. We estimate we’re going to knock down our development and [quality-assurance] environments by about 50%. It’s about 400 servers now.

Whose virtualization software are you using?

We’re using two pieces right now: Xen from Red Hat Linux and Sun Solaris Containers.

What other key steps have you taken to optimize your IT environment?

A big push that we’ve taken over the last couple of years is to really get off of proprietary hardware and get on to x86, Intel and AMD-based. We try to not focus on a single vendor. It could be HP, Dell or IBM. We don’t really discriminate. It’s what vendor can meet our specs and is giving us a good price.

Lots of people like to go with the same vendor to ease management, procurement and the like. How do you deal with some of those issues given that you prefer to use multiple vendors?

We have a spec that we wrote and have basically given the spec to all the vendors. We’re looking for them to design to that spec, which is along the lines of a Linux-centric server. What was hurting us was that a lot of vendors were taking the boxes that they were building for Windows and saying, ‘OK, you can start using it for Linux.’ And it was missing things like management, even things like hardware alerts. We work really close with the manufacturers and push them to get what we want.

How quickly do you amortize a typical system?

I think we were on a three-year cycle. But with our electronic trading environment it’s probably closer to 18 months, because when new hardware comes along, we can get benefit out of that. What we’ve found historically is that when we make the electronic trading environment faster, more trades happen. We’re not saying more trades happen because we’re faster; we’re just enabling it to happen. If each trade took a second, and you had one user putting in trades, they could get 60 in a minute. If you cut that down to a half-second, they could get 120 trades in. If your user base stays the same and your time stays the same, then you’re going to reach some limit as to how many trades you can have. Because we keep trimming the time back, we keep coming to a new record quarterly.

Desmond is events editor for Network World and president of PDEdit, an IT publishing company in Southborough, Mass. Reach him at paul@pdedit.com.