Riverbed wins 7-vendor WAN optimization test

Challengers Ipanema and Exinda score high marks for innovation

As applications move to the cloud, network managers are seeing increasing requirements to optimize and manage WAN connections. Most enterprises have migrated to web-based applications and make heavy use of Internet services for day-to-day business. All of this makes network performance a key factor for productivity and end-user satisfaction.

Vendors have responded with a bevy of devices aimed at improving network performance. Many focus on just one function, such as compression, bandwidth management or flow visibility. Our last test of network optimization products was in 2007, so we wanted to find out if things had moved forward. Could network managers improve WAN performance without buying a stack of diverse boxes for every office?

We invited every major network optimization vendor, and ended up with seven contenders:

1. Blue Coat Mach5 editions of the SG300-25 and SG900-10

2. Cisco Wide Area Virtualization Engine WAVE-7541, Cisco 4451-AX ISR and 2900-AX ISR.

3. Citrix CloudBridge 2000

4. Exinda Networks Model 6862 and 10862 running x800-series software

5. Ipanema Technologies ip|engine 1000ax and ip|engine 20ax

6. Riverbed Steelhead CXA-5050 and CXA-555

7. Silver Peak Systems VX-1000 and VX-5000. 

We put each product in the lab for an intensive round of testing in three key areas of network optimization: performance, visibility, and control. We also looked at evolving parts of the network optimization market, including application-layer controls and active data link management. Finally, we evaluated each product for its enterprise suitability, flexibility, and ease of use. 

[SLIDESHOW: Click through to see the pros and cons of each product]

Our Clear Choice Test winner is Riverbed, which excels at the core WAN optimization functions of compression and de-duplication. From a pure performance perspective, they know what they’re doing.  

However, WAN optimization has evolved to encompass other features, such as traffic management and visibility. Here, we find that Riverbed Steelhead could use some work. Traffic management is good, but not great. Visibility is limited in a way that pushes network managers to use Riverbed’s own tools, rather than opening up to the growing world of standards-based flow analysis products. And new features, such as WAN path selection, don’t live up to Steelhead’s traditional technology leadership.

If you’re looking for innovation, you’ll be as impressed, as we were, with Ipanema Technologies ip|engines and Exinda Networks x800-series. These two products offer a rounded approach to network optimization that we didn’t see in Steelhead. These vendors are clearly thinking beyond the basics to what the next generation of network optimization products should look like. As newcomers, though, we found glitches and holes in the products. For example, Exinda’s management system is weak, while Ipanema’s network integration and traffic management features are too rigid to work well with some networks.

For great performance, we were again impressed with Silver Peak, a top scorer in our 2007 test. Tied for first place in our compression and de-duplication tests, Silver Peak’s VX-series and NX-series simply makes things go faster. However, Silver Peak is having a hard time shaking their data center-to-data center heritage, and the VX-series and NX-series need updates to handle the requirements of networks with many branch offices.

Cisco’s WAAS product line is so broad that it’s difficult to know what to test, and then how to rank them. As Cisco WAAS moves from standalone devices to ISR-integrated software, we gain all the power, sound, and fury of IOS, but at a cost in complexity. If you’re happily committed to IOS on ISR devices at the edge of your branch network, adding WAAS is a no-brainer with big benefits at moderate cost. However, if you’re mixing Cisco with other vendors at the branch edge, the decision rarely weighs in favor of Cisco for network optimization.

We were disappointed with Blue Coat’s Mach5 and Citrix Systems’ CloudBridge 2000. Both cover the basics and work as advertised -- although CloudBridge 2000 probably needs some shakeout before being deployed. However, Blue Coat has failed to learn from its own technology experts how to integrate traffic management with compression. And CloudBridge 2000 only includes the barest of features required to compete in this area. Neither company is a standard-bearer for how to push a broad spectrum of network optimization features into branch offices.

Here’s a more detailed analysis of the products, broken up by functional areas:

wan optimization Network World

PERFORMANCE TESTING: Silver Peak, Riverbed come out on top

Performance is one of the first reasons network managers start looking at network optimization products, which use a combination of techniques, including caching (usually called “de-duplication” to distinguish it from the kind of caching that web proxies do), in-line compression, TCP and IP protocol optimization, and application-specific optimization. 

Each technique works in different ways, and, depending on your application mix, may be more or less beneficial. For example, if you move large text or database files around, in-line compression saves a lot of bandwidth. If you move a large data file that only changes a little bit each day, de-duplication helps. If your operating systems are tuned with small TCP window sizes or are using some types of congestion control, TCP/IP optimization helps. 

We decided to focus on the one feature that really counts for WAN network managers: the end-user experience.

We focused on five types of traffic that we thought would be representative of many enterprise WANs: encrypted and unencrypted web traffic, email, remote terminal (specifically Citrix Xen Desktop), and Voice over IP. We ran each test across five types of WANs, using an InterWorking Labs Maxwell link emulator to vary latency and loss to simulate traffic ranging from across a data center to through a satellite link.

We felt that high latency links were an important part of our testing. Pure WAN compression and de-duplication rarely pays for itself in areas where bandwidth is inexpensive and abundant — meaning much of the Americas, Europe, Asia and Australia. But when WANs are intercontinental, prices skyrocket and compression hardware can often be justified on a pure cost basis, ignoring other factors. 

We also decided that we could not fairly test CIFS file sharing, even though we know that many enterprises still use CIFS across their WANs. Because of the huge variation in CIFS optimization, we knew that our results would be very skewed based on small variations in how we tested, and we felt that publishing results on CIFS might be misleading. 

We started with HTTP and HTTPS traffic: internal web sites, SharePoint, web-based POS and ERP applications. Boiling down the hundreds of statistics available from Spirent’s WebAvalanche product, we focused on transactions completed: If we installed this product in this network, how much more work could get done? To do this, we compared the transactions we could complete on a fully utilized 45Mbps circuit with and without optimization. 

Here are the key findings:

HTTP TRAFFIC: Silver Peak and Blue Coat lead the way

  • For pure HTTP, Blue Coat Mach5 excelled in high-latency networks with an amazing 260% improvement in completed transactions compared to the baseline.
  • Silver Peak VX-series helped the most in low-latency networking, improving transaction count by 234%.  
  • Overall, every product except Citrix CloudBridge did well, boosting performance by at least 170%.

For the type of traffic that we used in our tests (a mix of HTTP objects), we think that most network managers will find that end users report the best performance with Silver Peak VX-series and Blue Coat Mach5 and slightly lower performance, but not significantly different, between Riverbed Steelhead, Ipanema ip|engine, Exinda x800-series, and Cisco WAAS.

wan optimization Network World

HTTPS TRAFFIC: Riverbed comes in first

  • Different vendors have different approaches to HTTPS, however, it’s pretty clear that the approach Blue Coat used wasn’t the right one, as it dropped from being the best performing in low-latency HTTP networks to the worst performing in HTTPS networks. Plug a Blue Coat Mach5 into a typical HTTPS session, and you’ll cut your performance by more than half.
  • Riverbed shot far in front of everyone else in HTTPS acceleration: an average of 185% improvement in transaction rate compared to non-optimized traffic.
  • Cisco WAAS and Silver Peak VX-series were next in line.
  • Citrix CloudBridge was at the back of the pack.
  • And Ipanema doesn’t claim to support HTTPS optimization at this point.

The vast disparity between HTTP and HTTPS traffic is an important factor for network managers to consider. If your traffic is all HTTP and HTTPS and performance is your number one criteria, Riverbed Steelhead certainly leads across both protocols, with Silver Peak VX-series a solid contender. 

wan optimization Network World

EMAIL TRAFFIC: SilverPeak edges the competition

  • When we tested email traffic, we found significantly less variability between products and protocols. The performance increases ranged from 140% of baseline to 166%.
  • Silver Peak VX-series beat the competition in email compression, both in low-latency and high-latency environment.
  • Blue Coat, Citrix, Exinda, Ipanema, and Riverbed are essentially indistinguishable, falling within a 5% range. 
wan optimization Network World

CITRIX TRAFFIC: It’s Silver Peak

  • We tested Citrix XenDesktop, the most popular enterprise remote desktop protocol for end-user applications, but didn’t expect to see much improvement because the Citrix application layer is already designed for WAN efficiency, and because Citrix XenDesktop includes both compression and encryption technology.
  • We weren’t surprised to find that Silver Peak VX-series led the pack in performance, but gave us only an 11% bump. 
  • Cisco WAAS, Citrix CloudBridge, Exinda x800-series, and Riverbed Steelhead essentially came in with identical performance of 102% to 103%.
  • Blue Coat Mach5 and Ipanema ip|engine trailed, in some cases actually causing a 2% to 4% performance decrease. 

Our results were based on our test scenarios, and Citrix is used in very different ways in different organizations.

In our testing, we didn't see serious benefits from compression. However, enterprises with many users at remote sites could get very different results because of the way XenDesktop packages traffic, which might benefit from the de-duplication in many of the devices we tested. Different usage patterns for XenDesktop and different user populations could have very different results. We think that network managers focused heavily on XenDesktop should still consider optimization devices like the ones we tested and either run their own tests or ask their favorite vendors for more information on performance tests based on XenDesktop. Optimization didn't help us, but network managers shouldn't write off optimization in every case.

wan optimization Network World

VOIP TRAFFIC: Silver Peak and Riverbed get the call

We looked at voice traffic (SIP-established Voice over IP) to see what happens with network optimization devices. Generally, the answer is supposed to be “nothing” for two reasons: VoIP traffic shouldn’t be compressible, because the CODEC in the VoIP phone has already done compression. Also, connectionless UDP traffic doesn’t fall into the “bump in the wire” model of the products we tested. To properly compress UDP traffic, the two devices would need to have an explicit tunnel so that they would know about the traffic, rather than auto-detecting compression via TCP options. 

  • Silver Peak VX-series doesn’t exactly operate in a bump-in-the-wire model, so it was able to get some performance gains out of our UDP traffic, an average of 9% improvement over baseline. 
  • Riverbed Steelhead, even without specific tunnel configuration, was also able to offer a 7% increase in performance. The rest of the pack turned in expected numbers, ranging from 99% to 103%.
  • However, these increases in performance came at some cost: increased jitter, which can be a serious problem for VoIP call quality. Both Silver Peak VX-series and Riverbed Steelhead kicked jitter up by about 10%. Our testing wasn’t able to include call quality scores, but, generally, more jitter is worse.
  • Citrix CloudBridge was able to reduce jitter across our test network by 16% compared to the baseline. 
wan optimization Network World

Looking across all protocols, Silver Peak VX-series and Riverbed Steelhead were neck-and-neck in performance improvement, with Cisco WAAS next, followed by Exinda x800-series and Ipanema ip|engine. Our testing found that in performance, Blue Coat Mach5 and Citrix CloudBridge were at the back of the pack.

TRAFFIC MANAGEMENT: Ipanema shines

If making applications go faster is one goal of network optimization, selecting which applications get priority, reserving bandwidth, and policing non-critical applications are all equally valid goals. We lumped those features under the banner of “traffic management.” 

1 2 3 4 Page
Editors' Picks
Join the discussion
Be the first to comment on this article. Our Commenting Policies