Beware product award inflation

As buyers, every one of us takes note of awards given to products we are interested in purchasing. Recent research I've conducted, though, shows that at least some magazine awards are given without the rigorous product reviews that one would naturally expect. Many readers take awards at face value and fail to dig for details. Those that take this approach may be in for a surprise if they buy based heavily on the "award count."

Product testing is rarely easy — and testing leading-edge, high-end devices such as intrusion-prevention systems (IPS) is a big challenge. Doing so can require both a mountain of expensive gear (from companies such as Spirent, Ixia, BreakingPoint and Mu Systems) and a mountain of expertise. A test can take weeks to set up and run. So how do magazines do it? Well, some skip the hard part — real testing — by just "kicking the tires" and handing out the awards. Caveat emptor.

In August, for example, a U.K. magazine published a review of an IPS that goes for $200,000. The box received an overall rating of four out of five stars. For the performance category it also achieved a rating of four stars with the comments "high performance, highly capable IPS… verdict: this is a serious IPS appliance meant for serious large-scale implementations." Strong words (and probably music to the vendor's ears).

To make this judgment, we naturally expect to see some "serious large-scale" testing, right? You are probably wondering what tests they ran, what test tools they used and what kind of performance numbers they turned in. I am wondering the same thing — and that is after reading the entire review.

To start, the review runs 397 words — shorter than this column by more than 100 words. For your convenience, I am now going to reproduce the entire performance testing section. Ready? "This device has no trouble with computing performance, but the protection performance sits right about in the middle of what we would expect."

That's it — one sentence. Summing up the performance of a high-end, $200,000 box in one sentence is a pretty neat trick. Numbers? Who needs numbers? The writer did say that it is "in the middle of what we would expect." Of course, he fails to mention what his expectations are. He also fails to note which tests tools and which tests were run. One can only assume that this doesn't matter.

An IPS has to scan and stop threats. Here is that section of the report: "On the scanning portions of our tests the appliance caught most of the attacks, and on the penetration attack it blocked everything from getting through." What attacks? Most? Is there a ban against using hard data — or did the reviewer realize that it is much easier and faster to write a review when you don't have to give specifics?

Ultimately a total of 48 words — the two sentences that I reproduced above — is given to the evaluation of the core functions of the device. And this is no anomaly; the same reviewer gave a "best buy" to a different vendor's $35,000 box with performance summed up in the single sentence: "This device performed very well throughout our testing." Indeed.

I'm sure that the vendors are happy to have found such an easy grader — but users will need to remain vigilant and know what is behind the awards they see.

Learn more about this topic

Blogs, truth and the enterprise

03/14/07

Wireless LANs and the security threat

02/20/06

Net infrastructure for the long run

08/07/06

Join the Network World communities on Facebook and LinkedIn to comment on topics that are top of mind.

Copyright © 2007 IDG Communications, Inc.