It’s called “The Cheater’s Guide to Network Testing” and I’ve been meaning to give it a plug since hearing it discussed at a trade conference and reading it shortly thereafter. Written by Timmons Player, a senior member of the technical staff at Spirent Global Services, the six-page white paper is designed to give network equipment buyers an inside look at the shenanigans — called specsmanship — employed by vendors and their marketing departments.
“Now, this article is not an attempt to discredit marketers — there are definitely some honest marketing organizations that will even understate their device’s capabilities in order to give themselves some headroom should a customer use a truly non-standard configuration,” Player writes. “However, this article is an attempt to enumerate the more flagrant forms of specsmanship and to educate the reader on these practices so that they will be able to recognize them in articles, data sheets, and wherever else someone has developed an insatiable love of large numbers.”
Among the topics the guide addresses:
— The purposeful redefining of common terms such as throughput, line-rate … and even megabyte;
— The largely meaningless exercise of quantifying device performance in terms of data rates absent any information regarding frame sizes;
— And the designing of tests that artfully tiptoe around architectural limitations of a device.
What was it that motivated Player to fire up his word processor? … The answer is disarmingly candid.
“I was prompted to write it after a particularly disappointing testing experience between Spirent, a nameless vendor, and a nameless third-party marketing organization,” Player tells me. “Essentially, the marketer and the vendor used about half of the tricks outlined in the article to dumb down a test to the point where it was impossible to actually know if the box they were testing was any good or not. Realizing what was going on and having to participate in it left me feeling pretty disgusting.”
“Additionally, I couldn’t think of any real way to combat the problem other than to educate the people who read the articles,” he adds. “Hence, I wrote the Cheater’s Guide to clear my conscience and let the casual equipment-review reader know some of the things that go on during marketing tests.”
Of course, one sure-fire way to avoid being sucked in by manipulative test results and the marketers who love them is to instead rely on the assessments of objective sources such as Network World’s Clear Choice Tests. But you knew that already.
The Cheater’s Guide is well worth a read.
