Watching over WAN data

WAN optimization shines at the U.S. Department of Health and Human Services.

Chris Finucane recently received an unsolicited e-mail from a user who wanted to know why his files were unexpectedly flying over the WAN. The question - which came from someone who often complained about network sluggishness - surprised Finucane. "If users notice the difference without you asking, you know you have something good," says Finucane, who is CTO in the Office of Inspector General for the U.S. Department of Health and Human Services.

In this case, that something is WAN optimization, an advanced technology for improving application performance over the wide area. A year ago, IT began deploying Riverbed Technology's Steelhead appliances in its 85 field and 10 regional offices, including its Washington, D.C., headquarters. In the process, IT has opened the floodgates for wide-area data transfers and positioned the OIG to handle bigger and more info-intensive projects. For these reasons, the HHS OIG is a 2006 Enterprise All-Star.

In search of better WAN performance

Finucane hadn't planned on exploring using this type of technology, already having allocated his annual IT budget to other projects. Plus John Rogosky, OIG's network manager, had just finished a network upgrade, swapping out 56Kbps frame relay links for T-1s between the offices and MCI's vBNS mesh network. But the bandwidth increase

didn't deliver the performance upgrade Finucane expected. "People should have been cheering. But not a lot of notice came out of that," he says.

The OIG is a watchdog organization for HHS' 300-plus programs, which include Medicare and Medicaid. OIG auditors make sure program funds are distributed and used properly; analysts evaluate HHS programs for efficiency and effectiveness; investigators look into possible instances of fraud or abuse; and attorneys provide legal services. OIG can assess monetary penalties for violations such as false billing, as well as exclude people from participating in federal healthcare programs.

With its investigative bent, the OIG is a data- and research-heavy outfit. While remote staffers had some access to centralized systems, they had grown accustomed to less-than-ideal e-mail performance and file server access over the WAN. With the bandwidth upgrade doing little to improve conditions, Finucane realized he needed to do more, especially as requirements for sharing data among far-flung users were increasing. "We had a couple of different initiatives that were pushing us to find a way to streamline our network," he says.

For example, OIG leaders wanted field staff to be able to query a central mainframe database, then produce and transfer customized test reports or spreadsheets. "With the existing infrastructure, it didn't look like we were going to be able to support it with an acceptable response time for our users," Finucane says.

In addition, the OIG was mulling long-range plans to consolidate its IT infrastructure. With its existing setup, it maintains servers in each of its 10 regional offices. This means carrying more user licenses than it would need if the applications ran on a central server. The OIG potentially could cut back on licensing by centralizing more servers and applications - but not unless Finucane could find a way to offset the performance hit that users in remote offices would suffer if they lost their local server power.

With so many network-intensive demands looming, Finucane and Dave Agsten, infrastructure architect at the OIG, started looking into ways to improve throughput. As part of that effort, the OIG tested WAN optimization devices from three vendors. The offerings were comparably priced, but Riverbed's Steelhead appliances delivered the greatest performance improvements, Finucane says.

Redundant transfers be gone

In tests and in production, Finucane found some of the most impressive gains came from Riverbed's ability to eliminate repetitive traffic from WAN links - a feature the vendor calls scalable data referencing.

Riverbed's devices work by intercepting TCP requests sent across a WAN link, then segmenting and indexing the data. Once the data has been indexed, it is compared with data on the disk. To speed transmissions, the appliance doesn't send data segments that already have been transferred across the WAN; a reference is sent instead. Even if a file is e-mailed first and later posted to a file server, the appliance can recognize the content and avoid sending it across the WAN.

The OIG tested the scalable data-referencing technology by sending 150MB files repeatedly. Data files previously processed by the appliance could be transferred in seconds, not minutes, even if the file name was different or some data elements had been changed, Finucane says.

The appliance also reduces the number of TCP packets required to transfer data. It intercepts TCP requests and repacks the TCP payload with references to data on the other end of the WAN, reducing the number of round trips generated by TCP. The Riverbed technology minimizes latency by reducing unnecessary chatter in applications such as Microsoft Exchange, he says.

Built-in management features let the OIG monitor system conditions, as well as track traffic reductions. The OIG sees the greatest reduction in Microsoft Windows Common Internet File Sharing transfers, but HTTP, Messaging API and SMTP traffic also are reduced. Overall, the data traversing the network is reduced to about one-tenth of what it would be without WAN optimization, he says.

Looking ahead, OIG is keeping tabs on Riverbed's plans to develop client-emulation software to let mobile staff achieve some of the same throughput improvements from laptops. The software would increase throughput and make even a dial-up line efficient for people on the road performing investigations and audits, Finucane says.

Finucane is glad he didn't go the add-more-bandwidth route. In retrospect, he says, OIG might not have experienced any improvement bumping up line speeds. And the expense would have been higher. "Doubling the T-1s in our field offices and adding more T-1s in our regional offices would probably have been the same as the cost of the first-year investment in the Steelheads," Finucane says. Because the OIG can avoid paying the cost of increased bandwidth going forward, that's money saved each year.