I am guessing (I can't tell for sure) that you're talking about the part of the test where we used the Mu-4000 to run various attacks through the IPS.

I think that perhaps we assumed too much in pointing to the UTM test---for example, that people would actually take the time to read the test or try and understand the comparison. The test we did for the Sourcefire IPS was the same that we did for the IPS built into the UTM firewalls. In that case, it's very much apples-and-apples, since the same criteria for accuracy would apply to an IPS, whether it was in a firewall or a standalone unit.

In the article's "how we tested," I mentioned that we used the Sourcefire provided aggressive policy, which Sourcefire calls "security over connectivity." I selected this policy because our testing over about a month showed a bare minimum of false positives, and thus this seemed to be a policy that would be appropriate to use.

I think that you might be making the same mistake you accuse me of by saying that the IPS blocked 95% of the attacks for you, especially when you don't mention the test tool used or even what the character of the test is. For example, if I used the set of attacks that ICSA uses, I would have had nearly a 100% catch rate! The point of the repetition of the test was to say that we had prepared some metrics on IPSes (in UTM devices) and it was a good idea to put this IPS to the exact same test. That gives people the ability to compare like products with the same test methodology.

I was pretty careful both in the UTM test and in this test to not say what these percentages mean; they simply allow you compare products using the same test. I would never expect anyone to have a 100% or even 95% catch rate using the Mu-4000 testing tool.