IPS systems gunned down on exploits ThreatEx fires at them.
With this test, we wanted groundbreaking performance measurements first and a comprehensive coverage and correctness assessments second. En route to those goals, we found the IPS systems stumbled so badly on stopping the exploits we threw at them during performance testing; we determined there was no point in proceeding with the coverage and correctness tests
While vendors tout their products’ ability to catch most known exploits, we found that even if we generously contend that an IPS might help block the casual, lazy attacker, they may not offer the necessary protection when faced with a more determined opponent.
Key vendors not in the game
Some of the biggest names in this space, including Cisco, Juniper and Sourcefire, were unusually gun-shy at the prospect of participating in our testing. The pressure point quickly became obvious. ThreatEx, the attack generator from our testing partner, Imperfect Networks (now owned by Spirent Communications), wasn’t a tool with which IPS vendors had much experience.
No IPS vendor will admit to tuning its product to perform well when tested with a particular tool. But the extraordinary sensitivity expressed by these vendors was a signal that we weren’t getting their products in our lab until they had a chance to run the tests themselves.
Vendors also asserted that any test tool could generate some traffic it claims will exploit a vulnerability — even if it doesn’t.They were concerned that ThreatEx would call them down for failing an invalid or incorrect test.
Imperfect Networks commissioned a third-party firm, System Experts, to validate the correctness of ThreatEx’s exploits. We also verified that each of the exploits we used resulted in a compromised system. Even with this intensive analysis and certification, we had some significant vendors stay away from our labs.
For our performance tests, we had to select some of Imperfect Networks’ attacks to represent the “bad traffic” to be part of the test. To make sure we could offer exploits at high rates, we selected three stateless, -based attacks: SQL Slammer, the Witty Worm and an attack against Cisco IOS (see Why no product stopped Cisco exploit). We assumed an IPS would catch these attacks, because all had been around for a while, have been widely publicized and attempt to exploit very common applications.
We were wrong. And not just a little bit wrong.For one of the exploits, a malformed Cisco SNMP query, all devices missed one form or another of the attack in initial tests. Some devices continued to forward the exploit even after we gave vendors subsequent opportunities for retesting.
Our original plan called for us to use about 100 different attacks in the correctness and coverage part of our tests, with another 25 or so “hard ones” thrown in just to see whether we could differentiate products. Here we had only three attacks, and 100% of the devices failed in one form or another. This wasn’t even supposed to be about correctness and coverage — we were working on performance.
Our experiences don’t speak well for the ability of an IPS to protect against a broad range of threats or against an attacker determined to evade the IPS. While the script kiddie with a well-understood exploit might get blocked at the door, our results suggest someone with inside knowledge and a moderate amount of access might well get by the guardians.
These results don’t mean that IPS systems aren’t useful at increasing the security of enterprise networks. What they do say, though, is that an IPS can be only one component in a defense-in-depth security strategy and that it’s much better to eliminate the vulnerabilities in your network by patching software and firmware than it is to depend on the IPS to provide protection.
Previous: Why no product stopped Cisco exploit | Next: How we tested IPS systems >
