New ways of benchmarking WLAN systems.
Benchmarking wireless products in any form is notoriously difficult, especially in comparative tests. The primary culprit is the behavior of the radio channel itself.
The first step we took to minimize variability was to monitor the radio environment. We used a spectrum analyzer to see a graphical representation of energy in the radio band being used. Good spectrum analyzers can cost $30,000 or more, and they require an engineer for successful operation in most cases. In our tests, we used AirMagnet’s A5120 Sensor, along with the company’s Enterprise 7.0 and Spectrum Analyzer. Enterprise was useful in visualizing 802.11 traffic, and the A5120 adds a chip from Cognio that let us analyze energy of any form in the 2.5GHz and 5GHz bands. Thus, we could find the best channel for our test (it turned out to be Channel 1), as well as monitor for any interference that might creep in (nothing interfered significantly).
In terms of antenna orientation, even with MIMO technology, it’s possible to place a notebook computer in a less-than-optimal position. In addition, people and objects moving in the environment can cause a fade. To account for these factors, we placed our notebook equipped with the MIMO PC Card on a turntable that revolved at 1 rpm – any orientation that resulted in fading would be very short-lived, and all products were tested under the same radio conditions. Power management was disabled on the rotating notebook; the power profile used was the same as when plugged in.
We performed two key tests on the MIMO-based wireless LAN (WLAN) systems: rate vs. range and interoperability.
Stretching the truth
Numbers that most vendors use, most notably along the lines of 240Mbps or 300Mbps, are not wrong, strictly speaking, but these are physical-layer signaling rates (akin to the 100Mbps of Ethernet). These vary significantly (upshifting and downshifting) during an individual test run. This is why it’s important to test Layer 7 application throughput and not rely on vendor claims of performance that can never be realized.
Rate vs. range
Unlike wire, wireless performance in terms of throughput can vary dramatically with range – the greater the range, the less the throughput. This is primarily because radio waves fade as they propagate through space, as signals fade exponentially with distance. Other forms of fading also may be present. One of these, Rayleigh fading, because of multipath, is less of a factor in MIMO implementations, as MIMO requires and depends on multipath for better performance.
We tested all homogeneous pairs of products (client card and router from the same vendor and product family) at a short (about 22 feet) and a long (about 81 feet) distance. Two PCs (a Panasonic Toughbook CF-74 and HP Compaq Nx6125 running Windows XP with Service Pack 2, fully patched) connected to the router generated the iPerf traffic. The PCs and router remained in the same location – a conference room with a large glass window and open door – for both distance tests. Two PCs are required, because the throughput available in a MIMO link easily could exceed the 100Mbps physical-layer data rate of a single PC connected by Ethernet.
The client location for the long-distance test also had windows and an open door, but a partially obstructed line of sight: Intervening space was occupied by cubicles.
Interoperability tests
Because there is no test for interoperability of MIMO-based products (at least when operating in full-speed MIMO mode), we felt it was important to examine interoperability in the light of vendor claims of draft 802.11n or “draft-n” compliance (see “The draft-n controversy.” In addition, we wanted to see what speeds users could expect for interoperability in real-world settings. Mixed-mode configurations are expected to be common until early 2009.
For heterogeneous products (different client and router), we tested interoperability and performance only in the short-range scenario.
Test procedures and objectives
Our core objective in setting our procedures for this series of tests was the application of techniques that enterprise IT staff could use themselves. We therefore used off-the-shelf tools, including the free Iperf LAN benchmar and Enterprise 7.0 as a configuration aid and monitor of 802.11 traffic and overall spectrum quality during all benchmark runs.
All the routers were set up manually; we did not use the CDs that came with each product. We checked vendor Web sites for the latest drivers and firmware for all products at the start of the project, and did not do any updates during testing. Client drivers were set up via the CD that came with each product, again checking for any software (drivers and client applications) or firmware revisions.
The only changes in router settings were for IP address, radio channel, Service Set Identifier and security (Wi-Fi Protected Access 2 with Advanced Encryption Standard encryption). We used client software provided by the vendors in all cases, which was better than the weak but convenient Zero Configuration tool built into Windows XP.
The project included more than 35 hours of testing. Anomalous results were investigated where noted, and replacement runs were attempted in these cases.
