How MSNBC beefed up its Web site for the Olympics

With the Olympics home page on its Web site, MSNBC is glad it pumped up its site for the onrush of traffic.

In a 48-hour period, a live vote on the figure skating controversy garnered more than 350,000 votes before it was pulled down off the site to allow for other Games-related votes. And on an average day before the Olympics, MSNBC had 4 million users per day and about 15 million page views. A tally from Feb. 12 showed 6.7 million users and 40 million page views that day alone.

Michael Corrigan, director of technology at MSNBC, decided last June that he needed to prepare for oncoming spike in traffic when the Winter Olympics got underway in February. Corrigan also realized the site could use a performance boost when user traffic skyrocketed in the days after the Sept. 11 terrorist attacks.

"In the news business, we need to run the Web site to handle whatever happens, whenever," Corrigan says. "And with the Olympics coming, we knew we had to get the infrastructure in place to guarantee our site."

Actually, Corrigan had contracted with two technology companies to ramp up the Web site for the Olympics -- but unfortunately, they both went bankrupt prior to working with MSNBC. Although he couldn't recall the companies originally contracted, Corrigan says he knew he needed to work with outside IT specialists to prepare the site in about six months.

"We knew we couldn't handle it internally. We needed help to determine what part of the system would fail and where we needed to invest," Corrigan says. Fortunately for Corrigan and MSNBC, KeyLabs, an e-commerce testing facility, was able to fill the void and optimized the site for Olympic-level traffic. KeyLabs stress-tested the site in October, using RadView Software's WebLoad load-testing software.

In January, MSNBC tested three sites with 100,000 concurrent users at a time. And the news site ensured that the three sites could support 2.5 million concurrent votes taking place in the online real-time polls.

The testing revealed to Corrigan that MSNBC needed to further invest in servers and software to get the site up to speed. MSNBC added 40% more servers to what the company originally had and purchased a front-end caching system from CacheFlow and load-balancing hardware from F5 Networks. The news outlet also moved some traffic from one data center to another.

"We knew we needed to fine-tune our caching systems, but the testing showed us things we didn't realize about the infrastructure," Corrigan says. Apparently, the tests paid off, because at 260,000 concurrent users - two-and-a-half times the normal traffic volume for MSNBC - CPU utilization is only at 20% with the requests per second topping out at 100.

"We managed to get the site ready in a cost-effective way. We didn't have to guess at infrastructure, and that helps," Corrigan says.