The big-picture view on product testing

Is it possible that security product reviews could actually be bad for the consumer in the long run? Interesting question. By focusing on things one can measure, and bypassing things one can't, one could argue that product reviews might encourage buggy, less secure products.

Most reviewers get a limited budget - in time, dollars and words - to evaluate a security product. In the space available, they usually focus on features and performance.

Reviewers like objective things, or nearly objective things. And measuring sessions or bits or megahertz is pretty darn objective. So is evaluating features. Runs on Linux, or doesn't. Scans for viruses, or doesn't. That's the kind of review that is defensible and fits in the constraints set for all of us.

Let's turn now to the product management team at a security vendor. They also have limited resources, such as an engineering team that can only do so much. Because reviews, press releases and product buzz are a critical part of any marketing plan, there's a lot of incentive to play to things to excite people. Lots of performance. New features. Whiz-bang graphical user interface tricks. So when it comes time to allocate engineering resources, product management is going to give a higher priority to things that help sell the product - and a lower rating to things that make it more secure but no one can measure.

A concrete example is stateful packet filters. Cisco, Check Point, NetScreen Technologies, SonicWall, WatchGuard Technologies - all use stateful packet filters inside their products. But none explains how they implement them. They have nice white papers with pretty pictures, but if you think about exactly what it takes to write a packet filter, there are many subtleties involved.

Packet filters follow the TCP state machine. The tighter the boundaries placed on the state machine, the less chance of a rogue packet making it through. You can follow the state machine very closely in writing your filter, or be sloppy. Taking the easy route has a lot of benefits. Your product runs faster. Your code is simpler and easier to write, meaning less chance of bugs. Plus, because you can hide behind the veil of trade secrecy, no one will know. Maybe a reviewer will figure it out. But it's unlikely.