Your mileage will vary

Factors listed in Clear Choice Test showing that individual cases differ.

Although some of our tests used actual content from popular Web sites, we’re not making any claims about our results regarding applicability in the real world. The term “real world,” with its implied prediction of actual performance in your network, glosses over many other variables beyond test traffic, including the mix of other applications in use, client and server hardware and software, and network design.

In a nutshell, “real world” usually isn’t. Rather than making any such claims about our tests, we’re instead doing the two things that are both possible and useful.

First, we take great pains to test each product the same way to the degree possible. This offers a starting point for comparing device performance – not necessarily performance in your network, but an apples-to-apples comparison all the same, at least as much as possible.

Even so, there are cases in which devices can’t be tested the same way. Not all devices we tested supported HTTP compression, and not all performed access control based on either source IP subnets or URL. There are other design differences as well (see One Size Doesn’t Fit All sidebar).

Second, we’ll freely acknowledge there are many factors beyond our test that could alter results. Perhaps the largest of these is that we did not attempt to test caching because of limitations of our test equipment.

Had we enabled caching, many tests – especially those measuring response times and transaction rates – might have been less stressful, because devices could have served objects from cache instead of fetching them from origin servers each time. Caching might also deliver a double benefit when coupled with HTTP compression, because devices could “precompress” objects and then serve them from cache.

Another factor to consider is the degree of compression, if any, performed by origin servers. In our tests, we compared each device against itself, once with compression enabled and again with it disabled. On the other hand, if servers already perform compression, a more meaningful comparison might be response times and transaction rates with and without a Web front-end device in place.

Other aspects of server hardware and network design can also affect results. For example, it may be useful to compare server CPU and memory use with and without a Web front end in place. These factors may also depend on the application(s) in use and the number of tiers used in the data-center design.

Numerous client-side factors also can affect test results. These include client-access network characteristics (including rates, delays, errors and packet loss), browser types, local browser caching, user think times and of course the number of users involved. Any one of these factors can have a huge impact on results. For example, our tests show that some devices are highly sensitive to user think times when it comes to TCP multiplexing efficiency.

When faced with all these variables, it’s easy to see how quickly “real-world” claims become meaningless. There is no one-size-fits-all approach that accounts for all factors in all settings. Our tests compare products in one setting, but we’ll be the first to acknowledge there are many other applicable settings.

How we did it | Next: Issues with compression >