Does Anyone Really Care About Benchmarks?

I spend a lot of time testing and benchmarking wireless LANs, in both individual and comparative exercises. I have always assumed that some form of testing and evaluation is a prerequisite for any major network-equipment purchase, and an enterprise-class, mission-critical WLAN certainly qualifies as major. As I've mentioned before, the fundamental costs and challenges inherent in large-scale WLAN testing provide significant motivation to find an alternative to this activity. But these also raise another question - looking ahead, will benchmarking itself remain important?

The reason I ask is that, in addition to the logistics of trying to do a meaningful and therefore valuable large-scale test, a great deal of time and (usually) dollars are also involved. And, given the vagaries inherent in the propagation of electromagnetic waves through space, the results we get might not be all that (a) accurate (and repeatable) and (b) useful in deciding whether to go with Vendor A or Vendor B. When we further consider the architectural differences between enterprise-class products (to say nothing of differences in radio implementation) as well as the hundreds of controller and AP settings inherent in each product that could have a more-than-statistical imapct on the outcome of benchmarks, freespace benchmark testing starts to look like a low-ROI activity. Besides, ultimately, won't all WLANs have either (a) roughly the same performance under similar operating conditions, and/or (b) more performance than we really need at any given moment in time? If (a), comparisons based on performance don't matter, and if (b) we're overprovisioned, and performance analysis, while interesting, again yields a low ROI.

I don't think, however, that we're at the point where either of these is true. We're learning more about how well specific WLAN systems handle specific operating conditions and traffic loads every day, often with surprising results. Last week I was invited to speak in a Webinar hosted by Veriwave, a company that builds test equipment for WLANs. They're branching out, however, to performance evaluation for enterprise end-users, as opposed to building products strictly for design and development engineers. It's possible to set up a large WLAN system configuration and test it using a synthetic workload that can be modeled to represent an anticipated (or any other) traffic mix. I call this virtual benchmarking, because it's not occurring in freespace, and it clearly has advantages in repeatability and cost. It's quite challenging, of course, to simulate the exact RF environment of any given structure, but at least we have a level playing field. Tom Alexander, CTO at Veriwave, noted during the Webcast that his products have already proven useful in finding bugs in products that otherwise would not have been discovered until long after deployment, and I have seen a correlation between real-world and synthetic benchmarks in some testing that I've done.

While I don't think that it can be definitively stated at this time that virtual benchmarking is a replacement for freespace testing, I think ultimately it has to be, again for simple logistical and economic reasons. I think that eventually we'll wind up with a standard set of tests for specific configurations that will, quite accurately, predict the performance of production installations well in advance of their realization. This is an area that deserves your attention - end-users will continue to ask which product performs best, and the test-equipment approach is likely going to be the most cost-effective route to an answer.