Testers' best tips

Network World Lab Alliance members share their secrets for conducting meaningful product tests.

By Paul Desmond, Network World, 02/23/04

 Start a discussion    Print article

 Comments (1)    Print article

Finding the products that will best fit in your network is a multifaceted process that involves scanning vendor Web sites, devouring white papers, attending trade shows and reading trade publications.

But once you come up with a short list, you might find that conducting a product test is the only way to make your final decision.

Product testing is something of an art, as we confirmed after polling the 14 members of the Network World Lab Alliance for tips on their trade. The lab alliance is the group of industry experts that conducts the product reviews you read each week in Network World. For this special issue, lab alliance members offer advice for your testing efforts that includes how to develop sound methodologies, acquire the gear you'll need and run tests that yield accurate results.

A valid test must meet three requirements, says David Newman, president of Network Test in Westlake Village, Calif.: It must be repeatable, stressful on the equipment or software under test, and meaningful. The last criteria is the most difficult to achieve, he says, but the secret is, "Test like you deploy and deploy like you test."

While that might seem like a tall order, the good news is that enterprise tests don't need to produce piles of data to be useful, says Joel Snyder, principal with Opus One in Tucson, Ariz. For example, a VPN test only needs to focus on two sets of numbers: performance using your typical mix of packet sizes and traffic types, and performance under a "worst-case" scenario, with peak traffic loads.

A method to the madness

Running a test that will get you meaningful results starts with creating a sound test methodology. Before devising a methodology, talk to peers within and outside your organization about how the product will be used and what features are most important, lab alliance members say.

Product vendors are another good source of methodology information. "More than once, feedback from vendor engineers has stopped us from doing something really stupid," Newman says, noting that any vendor's attempt to spin a test in its favor is typically transparent. He also says the IETF Benchmarking Methodology working group is a good source for methodologies and the Cooperative Association for Internet Data Analysis for measurement, performance monitoring, workload and other tools.

Related links

Network World Tests and Reviews
Our experts tell you which hardware and software is worthy of your net.

Network World Product Reviews newsletter
Get summaries of our latest reviews sent to your inbox every week.

About Network World, Inc. | Advertise | Careers | Contact us | Terms of Service/Privacy | Reprints and links | Partnerships | Press room | Subscribe to NW

Copyright, 1994-2008 Network World, Inc. All rights reserved.