1. Claim: Riverbed Steelhead appliance could not sustain rated capacity in WCCP deployment, consumed 90%+ of switch CPU, became unresponsive…

Miercom’s test purports to show that the Cisco WAE 7371 successfully ramps up to its fully rated capacity in a WCCP deployment while the Riverbed Steelhead appliance fails to do so. Vital elements of the WCCP configuration are omitted, but the test results strongly suggest that Miercom used a different WCCP configuration for the Cisco WAE 7371 than the one they used for the Riverbed Steelhead appliance model 6020.

In a WCCP deployment, a critical determinant of performance and switch CPU load is the choice of whether GRE or L2 is used for redirect and return of traffic. This aspect of WCCP on a Catalyst is likely what Miercom is referring to when they mention “hardware acceleration.” These parameters are negotiated as part of the protocol, and the default defined by the WCCP specification is to use GRE – which also imposes the highest overheads. Although Miercom does not provide enough information in the document to either confirm or refute this theory, it seems quite likely that this test simply exposes the difference in the switch’s performance in the two modes. Following Miercom’s rules as expressed in other reports, it’s OK to configure the Cisco appliance to use L2 and “minimally configure” the Riverbed appliance so that it uses GRE, even though such an arrangement renders the comparison meaningless.

2. Claim: Riverbed Steelhead appliance failed to pass-through mode on high traffic loads, in-path deployment…

For the second, in-path test, the Riverbed appliance appears to perform poorly – but the test is well beyond the appliance’s specified capabilities. The test generates 300 connections per second, each downloading a 512KB file. In a steady state, this arrangement has 300 connections starting each second while 300 others end. Simple math (300 connections/sec x 512KB/connection x 8 bits/byte) indicates a steady-state throughput of 1.228Gb/s of traffic was generated for this test. This is beyond what a single Steelhead appliance model 6020 would ordinarily be expected to handle, even on the LAN side. To support this environment, Riverbed would normally recommend a cluster of multiple Steelhead appliances.

In addition to running a test with too much data, Miercom also uses too many simultaneous connections. Careful wording hides this information from the casual reader, much as the report avoids stating the total throughput. The first paragraph describes this test as running “up to” 110% of rated capacity. This phrasing turns out to be either highly misleading or just plain wrong. Here’s why: the Cisco WAE 7371 has a specified capacity of 50,000 simultaneous connections while the Riverbed Steelhead appliance model 6020 has a specified capacity of 40,000 simultaneous connections. The description of the test as running “up to” 110% of rated capacity would likely lead a reasonable reader (and reasonable tester) to expect that the Cisco appliance will be driven to 55,000 connections and the Riverbed appliance will be driven to 44,000 connections.

Instead, a careful reading of the test results shows that the Cisco appliance was only driven to 49,997 connections (just barely under 100% of capacity) while the Riverbed appliance was asked to handle more than 60,000 connections (150% of capacity). Thus, the comparison is between a Cisco appliance at its specified capacity and a Riverbed appliance at an offered load of more than 150% of its specified capacity. Given the disparity of connections, it seems unlikely that the two devices were experiencing similar levels of throughput.

When confronted with traffic (both total throughput and connections) well beyond its designed capabilities, the single Riverbed Steelhead appliance performs as it should, which is to shut down operations in order to avoid adverse impacts to the network from the overload. This is the expected and desirable behavior for a device that is confronted with excessive demands from its environment.

For the only part of the test results that are comparable, they are actually quite favorable to Riverbed: the Riverbed appliance reaches its capacity and goes somewhat past it to 40,181 optimized connections in 3 minutes 45 seconds. Meanwhile, the Cisco appliance reaches 49,997 connections in 15 minutes (the only data point provided for Cisco in this test).

3. Claim: Riverbed Steelhead appliance was less effective at optimizing CIFS traffic than Cisco WAE…

Miercom claims that by running a test script of file operations, they proved that Riverbed was less effective than Cisco at optimizing CIFS traffic. Cisco has made this claim without any effect since September 2006, and Riverbed has previously refuted it; nothing has changed in the product since then to make it any more true. For completeness, we repeat the previous explanation here.

The claim of additional server load is misleading, although there are two grains of underlying truth. First, adding clients in a Riverbed environment will add load to the server, just as if the clients were added in the server's LAN environment. Second, read-aheads that are unneeded are an additional load on the server. However, we have yet to find any customer for whom either of these issues has been a problem in a real deployment. Read-aheads place a small additional load on the server when data is read that is not needed, but the majority of the read-aheads are used by the clients.

Miercom’s test simply exploits an architectural difference between Cisco’s approach and Riverbed’s. Riverbed has never claimed to offload servers in the data center, and in an environment like this test – one server, no concurrent usage, and likely a heavy bias toward read operations – it is not surprising that a cache would look good. The problem is that a cache’s apparent virtues in simple cases become liabilities at large scale or in complex cases. Riverbed keeps the server in charge of what data means and what data is served out while still providing excellent performance (as acknowledged by Miercom).

Any cache architecture, Cisco’s included, necessarily puts customers into the difficult situation of understanding when a server will serve data vs. when a cache will serve data. That complicates issues of access control, authentication, sizing, and security, as well as introducing challenges of ensuring that data is consistent.

4. Claim: Riverbed Steelhead appliance was less reliable than Cisco WAE…

Miercom claims that the Riverbed appliances “exhibited frequent and repeatable instances of system instability, false alert conditions, and system crashes when subjected to heavy load.” They also claim, remarkably, that “the test load conditions were well within the specified capacity of the product.” This last claim is easily refuted by simply doing the math on their reported load-test results: either their results are reported incorrectly, or the claim about staying within Riverbed specifications is incorrect, but there is clearly an internal contradiction in the report.

A screen shot of a Riverbed Steelhead appliance shows numerous alarms, and is intended to convey that the system is unstable. However, a careful examination of the screen shows that Miercom took some action to cause a rebuild of the RAID cluster. An identical screen can be produced by deliberately removing a disk drive from a running system. The screen is almost certainly not from any overload test that they conducted, since a system passing through excess connections would show an alarm for “admission control,” not seen in the screen shot. Without additional detail from Miercom, it seems likely that this screen shot just looked scary and is unconnected to the load tests described in the report.

5. Claim: Riverbed Steelhead appliance was less efficient and “green” than Cisco WAE…

Miercom notes that the WAE-7371 is a 2U device while the Steelhead appliance model 6020 is a 3U device, which is unquestionably true.

Intriguingly for a testing company, Miercom did not actually test any actual power consumption or cooling characteristics of the products. Instead, they claim to have relied on publicly available information. However, they did not rely on the actual published data for the Steelhead appliance’s thermal characteristics. The Riverbed Web site includes a specification sheet (full URL at end) that lists the Steelhead appliance model 6020 at 1450 BTU/hour and 425 watts. Miercom instead claims values of 5900 BTU/hour and 1140 watts. Miercom apparently got 1140 watts by simply taking a single power supply maximum (380 watts) and multiplying by three for the use of three power supplies. In fact the Steelhead appliance model 6020 does not need to use more than two power supplies, and does not require maximum power from those two supplies. Riverbed’s published numbers already incorporate the fact that the Steelhead 6020 has three power supplies, and would have been the right basis for comparison.

Cisco’s hardware installation guide for the WAE 7371 (full URL at end) also gives different power and thermal numbers than the ones Miercom used. We’re not sure why the discrepancy arises, but Miercom actually uses a worse heat output number than the official Cisco specification (Miercom says 4350 BTU/hour, Cisco says 3390 BTU/hour). There seem to be mathematical errors in all of these numbers, since none of the BTU/hour numbers are very similar to the watts numbers when simply converted. However, giving the benefit of the doubt, we can pick the smallest Cisco/Miercom number of 830 watts (which converts to 2830 BTU/hour). We then note that the WAE 7371 power consumption is approximately twice that of a Steelhead appliance model 6020 – not, as Miercom claims, a small fraction. In addition to its other virtues, the Riverbed product is more efficient and “green” than the Cisco product.

Specific questions that I would like a reply to from the Miercom.

1. In the WCCP test, are the Cisco and Riverbed units being tested with the same configuration of redirection and return, or is the Riverbed test using the less-efficient GRE mechanism?

2. How many connections were in the in-path workload for Riverbed? How do you explain the discrepancies in different parts of the test description between claims of 110% of rated capacity (44,000 connections), 40,000 connections, and the actual connection counts shown of nearly 60,000?

3. Why is there no in-path test result showing a Cisco WAE 7371 running at or above full capacity?

4. What happens to a Cisco WAE 7371 when you drive it to 150% of capacity, as was done for the Steelhead 6020?

5. Where did Miercom get their power consumption numbers, since they claim to have used publicly available data but the numbers don’t line up with either vendor’s specifications?

General questions that I would like a reply to from the Miercom.

1. Shouldn’t Miercom document their test setup and their data in enough detail to allow vendors and customers to replicate their tests?

2. Why does Miercom use different testing methodologies for different WDS vendors?

3. Of Miercom’s four clients, which WDS system is actually the best?

4. Why does Miercom think Miercom is repeatedly paid to test different vendors against Riverbed?

While we are not saying which is better or promoting either vendor, we are just presenting the facts that they left out.

Sincerely,
The WDS Underground.

Of course that Miercom benchmarks (as well as Tolly and others) are biased...

Any test can be tweaked to show that one vendor is better than the other, my biggest concern with both of these vendors, is that they are both severely limited! (especially for such EXPENSIVE devices)

lets do the simple math here, 40K or 50K optimized TCP sessions - hmmm what does that really mean? well in my case 40K sessions represented roughly 2000 users (just run netstat on your computer and see how many sessions you are using...)
Of course if you ask riverbed they say that you should expect 5 TCP sesssions per user - just outlook uses 13 session on my machine for some odd reason, Cisco on the other hand, after some arm wrestling, will admit that 10-15 is more reasonable assessment, however the information on NetQOS showed that we had between 18 and 22 TCP sessions per user.

We recommended a different vendor to the CIO (just look for the one that supports 256K TCP sessions), but he went with the "CIO friendly" option - Cisco (no one gets fired for buying Cisco...) in the mean time 6 months have passed, and we still can't get the Cisco gear to work, as their own PIX firewalls don't allow WAAS traffic through (don't get me started on that...)

Of course that Miercom benchmarks (as well as Tolly and others) are biased...

Any test can be tweaked to show that one vendor is better than the other, my biggest concern with both of these vendors, is that they are both severely limited! (especially for such EXPENSIVE devices)

lets do the simple math here, 40K or 50K optimized TCP sessions - hmmm what does that really mean? well in my case 40K sessions represented roughly 2000 users (just run netstat on your computer and see how many sessions you are using...)
Of course if you ask riverbed they say that you should expect 5 TCP sesssions per user - just outlook uses 13 session on my machine for some odd reason, Cisco on the other hand, after some arm wrestling, will admit that 10-15 is more reasonable assessment, however the information on NetQOS showed that we had between 18 and 22 TCP sessions per user.

We recommended a different vendor to the CIO (just look for the one that supports 256K TCP sessions), but he went with the "CIO friendly" option - Cisco (no one gets fired for buying Cisco...) in the mean time 6 months have passed, and we still can't get the Cisco gear to work, as their own PIX firewalls don't allow WAAS traffic through (don't get me started on that...)

Dear writer,

It's really sad that obviously sponsored reports like this
have no value for the reader. Nice that you make some money with it but please spare us with your greedy style of journalism.

To the networkworld team i recommend reading all article first before publishing it as reports like these will damage your reputation in the market.

I appreciate product comparison but please keep them objective.

best regards

Interesting that only pay-to-play Miercom publishes favorable Cisco WAAS tests. Any product can be made to look good (or not so good) in a lab

The reality is, in the real world, Riverbed's win rate vs Cisco in customer environment bakeoffs is over 90 percent!!

Analysts like Gartner and Forrester and product tests from Infoworld and network consistently hail Riverbed as the performance leader by far.

Over 3000 Cisco network customers and counting selected Riverbed to go faster? Wouldn't it just be easier to add to their Cisco infrastructure? Obviously not.

The proof of the pudding is in the eating