With this test, we wanted groundbreaking performance measurements first and a comprehensive coverage and correctness assessments second. En route to those goals, we found the IPS systems stumbled so badly on stopping the exploits we threw at them during performance testing we determined there was no point in proceeding with the coverage and correctness tests. 

While vendors tout their products’ ability to catch most known exploits, we found that even if we generously contend that an IPS might help block the casual, lazy attacker, they may not offer the necessary protection when faced with a more determined opponent.

Key vendors not in the game

Some of the biggest names in this space, including Cisco, Juniper and Sourcefire, were unusually gun-shy at the prospect of participating in our testing. The pressure point quickly became obvious. ThreatEx, the attack generator from our testing partner, Imperfect Networks (now owned by Spirent Communications), wasn’t a tool with which IPS vendors had much experience.

No IPS vendor will admit to tuning its product to perform well when tested with a particular tool. But the extraordinary sensitivity expressed by these vendors was a signal that we weren’t getting their products in our lab until they had a chance to run the tests themselves.

Vendors also asserted that any test tool could generate some traffic it claims will exploit a vulnerability — even if it doesn’t.They were concerned that ThreatEx would call them down for failing an invalid or incorrect test.