Testing .11n - Again

OK - the article I wrote on the latest round of .11n tests is out, and you can find it here. It's amazing how much time goes into projects like these - more than two weeks to test five routers and five client devices. Of course, there were also two baseline tests (one .11g combo and last year's non-Draft n winner, tests of every combination of router and client for TCP throughput, and also all homogeneous pairs (client and router from the same vendor) on UDP throughput and video performance as well. Benchmarking is meticulous work - checking installations, verifying configurations, running tests, monitoring for interference, noting anomalies - I even had to start over, from scratch, after discovering a bad cable (imagine that!) between the router and the PC running the Iperf client. It's always the little things, but uniformity is essential to meaningful results.

The objective of comparative benchmarking is to learn how different combinations of products respond to the same workload. I have been running benchmarks on WLANs for many years, and I think I've developed a methodology that works very well; you can read the details in the article. I first got interested in benchmarking .11n when the first "Draft n" products, based on the first draft of the standard (yes, the one with a record 12,000 comments) of early last year. I was at that time the wireless columnist for Electronic Engineering Times, and Patrick Mannion, then Editor-in-Chief, called and said he could get me a big empty house, and all I had to do was get the .11n products and run a bunch of tests. And, oh yes, I only had two days over weekend to do the testing and that would be in a couple of days. Of course, like any nerd, I said absolutely, and off I went.

What I discovered was a serious lack of interoperability among the products tested - no surprise, really, since there was no one certifying interoperability at the time. Note that the IEEE does not certify conformance or compliance to its standards, and also notes, in bold type, that it frowns on claiming compliance with drafts. The whole Draft-n thing, at least with respect to Draft 1, had become a serious boondoggle. MIMO products not claiming Draft-n compliance and preceding Draft 1 had much better performance regardless, and it was clear the marketing departments at the vendors were suffering from irrational exuberance. The subsequent testing for Network World noted above showed the same thing - poor interoperability, and tests published in other magazines confirmed the pattern. Don't' get me wrong - the Draft 1 products were still a lot better than .11g, but interoperable? No way.

OK, now we have Draft 2 of the proposed and still-under-development .11n standard, and the Wi-Fi Alliance is certifying compliance. But all is not peachy and rosy at this point. We're still seeing incompatibilities that negatively impact interoperability. A few products need a little more time in the oven, especially with respect to drivers. But, overall, progress.

Let me know what you think. As the sidebar to the article says, I still believe performance testing of this form is important. And the issues and opportunities we've seen with the residential-class products we've tested are likely to carry over to the upcoming (many announced already) enterprise-class .11n products.

• 802.11n
• benchmarks
• Draft 2
• testing