We tested the intrusion prevention capabilities of each of the next-generation firewalls to determine how well they work and how the IPS integrates with system management.
We were especially concerned with the IPS workflow for false positives, taking the network manager from a logged IPS event to the particular IPS signature triggering the event, to the ability to disable or modify the IPS signature to reduce problems.
We started by using our Mu Dynamics Studio Security test tools to check how well each firewall's IPS would catch Mu's list of published vulnerabilities. We tested the firewalls in two different configurations, one optimized to protect end users, and a second one optimized to protect servers. For each configuration, we sent a different set of about 1,000 vulnerabilities.
For each vulnerability set (server attacking and client attacking), we created two policies for each firewall. One policy included all of the IPS signatures and the other just had the subset of signatures marked as highest priority. We were thinking that the "all" signature set would have more false positives, and most network managers would want to only block the most critical vulnerabilities.
In most products, we saw less than two percentage points of difference between the two sets, meaning that there's very little tuning of the IPS possible. Fortinet's FortiGate was the exception, showing a 10% to 25% difference in attacks blocked, offering the network manager more tools to match the IPS to their network.
When protecting clients, we found that the Check Point Security Gateway, Fortinet FortiGate, and Barracuda NG firewall all outperformed SonicWall SonicOS. However, when we tested server-protecting IPS configurations, SonicWall and Fortinet performed significantly better than Check Point and Barracuda.
We believe that most enterprises deploying next generation firewall functionality will be doing it to protect end users rather than servers, so the client-protecting IPS coverage is more important than server-protecting coverage.
While we think that testing with the Mu Dynamics tester helps to keep IPS vendors on their toes with vulnerability signatures, it's important not to read too much into efficacy tests like these. Since the Mu Dynamics tester is a standard product, there's always the possibility that IPS vendors will tune their systems to increase their scores — even if they don't agree that a particular attack or vulnerability is important or correctly crafted. The Mu Dynamics tester is also useful because it can do mutation testing, which can stress the software in next generation firewalls, although only the Barracuda NG Firewall had a crash during our test runs.
Because IPSes all can trigger false positives, management is an important concern. We found Check Point Security Gateway and SonicWall SonicOS the easiest to work with, although this can be a matter of personal preference. Both devices only allow for a single IPS policy per device, which means that you're managing a single large policy on the firewall. That's limiting, but it is an appropriate limitation when you're managing a firewall and not a dedicated IPS device.
In contrast, Barracuda NG Firewall and Fortinet FortiGate both allow you to define multiple policies, and bring each policy into play on a rule-by-rule basis. The NG Firewall and FortiGate are more flexible, but there's a price to be paid -- you don't have very good policy creation and management tools, which means that making more than one policy can turn out to be just aggravating.
If you think your IPS management will be a "set it and forget it" style where you define rough categories you want to enable and then never look again at the logs or the configuration, you'll be happy with any of these products.
When we turned to the IPS reporting interfaces, we found a clear winner in Check Point's Security Gateway when combined with the optional SmartEvent analyzer. Check Point's winning combination offers an easy-to-understand way to view IPS events, understand what is happening over time, and to drill-down into individual events and supporting evidence for each event. From the SmartEvent analyzer, we were able to jump directly to the IPS policy, enabling or disabling signatures or adding exceptions.
If you are thinking of replacing your standalone IPS with a next-generation firewall containing an IPS, and want to have the same level of reporting and analysis that a standalone IPS gives you, Check Point Secure Gateway with SmartEvent analyzer leaves the other devices far behind.
Second place in IPS reporting goes to SonicWall, when combined with their optional Global Management System. Fortinet FortiGate and Barracuda NG Firewall both were in our labs without separate reporting systems, leaving only the on-box analysis tools. FortiGate offers a nice slate of on-box IPS reporting features including some drill-down capabilities, but didn't do as good a job of presenting IPS information as either SonicWall Global Management System or Check Point SmartEvent analyzer.
Overall, the next-generation firewall closest to a standalone IPS in its visibility and policy management capabilities is Check Point's Security Gateway (but only when combined with the SmartEvent analyzer).
If you just want a set-it-and-forget-it IPS, Fortinet barely edged out SonicWall and Barracuda by bringing in a higher catch rate in our Mu Dynamics' IPS vulnerability coverage tests.