Testing taboos

Asked what mistakes are most common in the testing business, Lab Alliance members came up with a hefty list, including these.

John Bass, Centennial Networking Lab, NCSU

• Getting too caught up in the details. (Relax, it’s just a box.)
• Not documenting the details of methodology and findings; repeatability should come first.
• Trying to test with only one method. It helps to test the same function of a box in different ways.
• Trying to draw too many conclusions from results.

Christine Perey, Perey Research & Consulting:

• Not having a clear, written test plan. You always should have hypotheses you are testing and specific usage scenarios.

Ed Mier, Miercom:

• Recording results incorrectly.
• Trying to be too fast.
• Reading too much into a set of results data.

Greg Goddard, EDS:

• Not reading the manual.

Jeffrey Fritz, University of California, San Franscisco:

• Not understanding how a user will deploy the product under test.

Thomas Powell, PINT:

• Letting vendors get too involved. Typically customers don’t have access to vendor CTOs.

Thomas Henderson, ExtremeLabs:

• Not assessing product administration, which plays an important part in the overall cost of ownership of a product.