How usable are network intrusion-prevention systems?
The most important feature of an IPS is whether or not it does the job you bought it for. That said, it also needs to be usable in the sense that it can support the network manager in the day-to-day tasks that go hand in hand with using an IPS in an enterprise setting. After shaking out the IPS products for performance, we took them back into the test lab to look at them from another angle entirely: usability.
Reviews of 6 products: Ambiron | Demarc | Fortinet | NFR | TippingPoint | Top Layer
Archive of Network World tests
Subscribe to the Network Product Test Results newsletter
On the dark side of our scoring, though, were the management wares provided with Demarc’s Sentarus, Fortinet’s FortiGate 3600 and Ambiron TrustWave’s (formerly Lucid Security) ipAngel. While each of these three has its administrative bright spots, all three need substantial work before they can handle the tasks we think concern an IPS manager.
However, it’s important to keep in mind that both Demarc and Fortinet offer multifunction products (the Fortinet box is a UTM device, while the Demarc product is a combination of host and network-based IPS), where network-based IPS is only a piece of a bigger offering. Network managers may be willing to trade off IPS usability and features in exchange for the other security functions shipped with these products.
We set up a between our test labs in California and Arizona to see how these products would work in an enterprise WAN environment. Because all of the performance testing was done at Network Test’s lab in California, we did all our usability testing from Opus One’s lab in Arizona. Where vendors provided stand-alone management tools for their products, we used those tools. Otherwise, we used whatever native tool was built into the IPS itself.
To evaluate products for usability, we set out five major task areas all geared toward operating an IPS in an enterprise-class network. We started looking at configuration and alerting capabilities, because these are the first tasks any security manager will do and subsequently be revisited repeatedly as networks, systems and security policies change.
Next, we looked at the dashboard features for each product, to see how easy it is to get an update on the security status of your network. We looked carefully at forensics features. Although an IPS is not a replacement for an , most IPS products have a fairly strong set of IDS forensics and analysis features in them, and we wanted to see how well this was put together. Finally, we looked at the reporting features of each product.
Task 1: configuration
IPS configuration can be an ugly and complex job. To minimize false positives and maximize coverage, the network manager must push a considerable amount of network topology and configuration information into the IPS. For example, the IPS configuration may need to include signatures specific to the ports being used (such as 80, or 8080), Web server (such as IIS or Apache), application languages (such as PHP) and third-party applications (such as calendaring or discussion forum software). Without this level of specificity, the IPS manager risks either missing attacks (for example, by not looking at traffic on all the right ports or not having the right signatures turned on) or having false positives (for example, by having signatures on that don’t apply).
Most IPS products take an IDS-like view of things and encourage the network manager to turn on all signatures on the common ports, such as Web traffic on port 80. That makes configuration easy, but it guarantees that the IPS is going to perform in a suboptimal way. System throughput and latency will be affected by having extra signatures looking at traffic, and any unusual network configuration will open up a hole for attackers that the IPS may not be looking for.
Configuration also goes beyond turning signatures on and off. Many of the IPS products we looked at, especially those from TippingPoint and Top Layer, have extensive rate-base IPS features that need to be tuned to be effective. In addition, an IPS could have different actions (other than “drop the packet”) when a signature is triggered. For example, some network managers want to drop all future traffic from an IP address once the IPS has alerted on any attack.
Our testing showed that TippingPoint configuration interface offers an excellent model for defining configurations, based on rule sets. TippingPoint’s Security Management System’s ability to give per-rule granularity, to version-control sets of rules and to apply globally defined sets of actions and alerts to rules and groups of rules really makes it easy to define and control your configuration. We found some things to pick at, such as the lack of rule-grouping tools, but this product demonstrates that TippingPoint really knows what network managers need in an IPS configuration tool.
If rate-based IPS is a priority, then the Top Layer IPS 5500’s configuration hits the top spot. While the TippingPoint 5000E does have some rate-based IPS features, no one has Top Layer’s precision and control when it comes to configuring rate-based rules. Top Layer’s signature configuration is not as strong, reflecting the roots of this product in the rate-based IPS world.
This brings us to NFR’s Sentivist combination, a product set that has to be faulted by a most network managers for being too granular and having too many options. Fundamentally, Sentivist is really an IDS, not an IPS, reflecting its roots as a capture-and-analysis tool for network traffic, and this comes through in its configuration options and management style. It’s easy to describe who would prefer the NFR approach: someone who really wants to control exactly what signatures will trigger an event on their network, to know how the signatures work (since NFR includes source code for most parts of the signatures), and to have the ability to tweak and adjust every aspect of the signature.
NFR’s Sentivist Management Platform is a tool for someone with a doctorate in network security who has the time and interest to spend hours a day getting everything perfect. For example, while most of the IPS systems we tested may have a dozen or so settings that are used to tune signatures, NFR has hundreds of variables to tune and tweak its system. The network manager who wants a dual-purpose IDS and IPS, and who intends to really use the tool, should put the Sentivist at the top of their short list.
We had mixed experiences with Ambiron TrustWave’s ipAngel product in the area of configuration. The ipAngel management code we received was a late beta version, and we ran into a number of bugs that hampered our usability testing of its interface. What ipAngel does that is different from other IPS products is closely tie vulnerability information gathered by the ipAngel device to the actual signatures in use on the IPS. Thus, ipAngel runs a scanner against your network and decides what systems are there, what ports they are using and what applications are running on those ports. Then, it can automatically turn on the rules to tightly tune the IPS just for those systems.
All of that sounds great, except that the code we got didn’t work that way. The scanner wouldn’t launch; the rules couldn’t be edited; the resulting configuration was locked in and couldn’t be changed. Network managers who want signatures on even if the host isn’t there (for example, to handle lab environments or systems undergoing rapid reconfiguration) had no way to do what they wanted — in the version that we looked at. Conceptually, ipAngel looks like it has some great ideas in it, but didn’t quite get everything finished in time for us to test. We were also hampered in our testing, because the ipAngel had no documentation or online help.
Network managers who are primarily interested in a network-based IPS will be disappointed in the configurations tools in supplied with Demarc’s Sentarus and Fortinet’s FortiGate 3600. Both have only the most rudimentary IPS configurations. For example, in the Sentarus product, changing an IDS rule to an IPS rule requires that you bring up each rule, one at a time, and edit it by changing the word “alert” to “block.”
FortiGate’s configuration has its own restrictions. For example, rate-based IPS settings are defined globally, which means that you can’t have different thresholds for the Web server handling 5,000 pages per second from the one handling five transactions per minute. We also found the documentation on signatures to be so incomplete and misleading that we were unable to understand what many of the signatures in their database actually do, or what vulnerability or exploit they are testing for.
Task 2: alerting and actions
An IPS has to be able to drop suspect traffic, but the difference between a basic and an enterprise-class IPS lies in the capability to do more than drop packets. This is where alerting capabilities and notification controls come in. While these abilities are always present in an IDS, we believe that high-end IPS systems should also have the ability to deliver alerts and take actions during ongoing attacks.
As an aside, it is important to note that in the world of IPS and IDS, the term “alert” has several meanings and is sometimes used interchangeably, albeit confusingly, with the term “event.” An “event” occurs when a signature is tripped, which might cause an action such as dropping a packet or sending a message. In this article, we’re using the term “alert” to mean a higher-level message, such as an e-mail or page, coming out of the IPS-management system in response to one or more events. Typically, every event will be logged unless you specifically suppress logs for uninteresting events, but higher-level alerts would be generated only for more critical events.
The job of alerting in a large IPS deployment may not fall upon the IPS, though. Network managers who have Security Information Manager or Security Event Manager (/SEM) tools may want the IPS to pass event information onto their SIM/SEM and let it make alerting decisions at a more global level, perhaps based on correlations across devices or even different types of security devices.
Network managers who care about alerting in the IPS itself will want to start with TippingPoint’s 5000E, although Demarc’s Sentarus and Top Layer’s IPS 5500 came in right behind TippingPoint in our assessment. In the TippingPoint 5000E, the network manager defines “action sets” that can be associated with signatures. An action set can be extensively customized. Actions can include dropping traffic, blacklisting attackers, rate limiting, connection resets, and a variety of alerting and logging actions, including capturing packet traces.
What impressed us most about TippingPoint’s alerting capabilities is the obvious ease of applying these actions to different signatures and signature groups. Rather than digging around through different parts of the GUI to figure out where to apply these actions or having no granularity to give different actions to different signatures, TippingPoint really hit the nail on the head with this design, combining ease of use with power in a way that makes it easy to get highly granular alerting when you want it.
Demarc’s Sentarus has another view of alerting that proves some great thinking. Rather than apply alerts to signatures, you define filters that watch the stream of attack events. The powerful feature of Sentarus that makes it act more like a SIM/SEM is the ability to filter any field in the attack. For example, you could place a alert on any attack event that is aimed to or from a particular IP address.
Unfortunately, the Sentarus developers left out any easy way to do the simpler task of applying and managing alerts attached to a signature or signature group. Demarc told us that the company plans to add more signature grouping facilities in a future release of the product, which would help with this.
With Top Layer’s IPS 5500 alerting capabilities, again the rate-based IPS features trumped signature-oriented IPS features. In the rate-based portion of the IPS 5500 5500, the alerting is excellent and provides a variety of thresholds and options to let you know when something is wrong. On the signature-based part of the IPS, the capabilities are extremely limited.
You could argue that alerting is much more important when it comes to rate-based IPS than on the signature-based side of the product. After all, rate-based IPS attacks are long-term events and may require some manual intervention or mitigation, so immediate alerts are pretty important. For signatures, most alerts are really confirming that the IPS has done its job, and that might be a job better suited to reporting.
NFR has built some SIM/SEM features into Sentivist that the company refers to as “correlators.” A correlator is a filter that watches attacks and generates events only when some condition occurs, such as a cluster of five attacks from the same IP address. The most interesting correlator we found was one that takes the results from Tenable Network’s Nessus vulnerability scanner and uses this information to generate an alert only when an attack matches a known vulnerability — behavior similar to Tenable’s Security Console (see test of Tenable product suite).
Unfortunately, NFR didn’t apply the same powerful thinking when considering how to deliver these alerts to the network manager, and put in place what amounts to a do-it-yourself toolkit for alerting. This matches nicely with the general NFR philosophy that you have to be really smart, really experienced in network security, and really enjoy Unix system management and programming to really use their product to its full potential.
In Fortinet’s FortiGate 3600 and Ambiron TrustWave’s ipAngel, alerting is more of a global setting than something that is related to individual signatures. ipAngel was particularly weak in this area, unable to even send e-mail when something went wrong.
Task 3: quick status and dashboard
Whether you want to spend time paging through the event messages generated by your IPS, you’ll almost certainly want to have some way to get an instant status report on the security of your network. A status report or dashboard reading would typically include some way of graphically representing trends, problems and traffic levels. A good dashboard gives you the ability to see in seconds whether things are at their normal levels or whether something bad is going on that would require your immediate attention.
Demarc’s Sentarus, NFR’s Sentivist and TippingPoint’s IPS 5000E all have good dashboards. For example, TippingPoint’s dashboard includes nine separate windows; each includes such features as minireports on top attack signatures, systems being attacked or a graph showing attack rates. TippingPoint includes a quick system health status on the same page, something we think is pretty important now that we know how easy it is to drive these IPS products to their performance knees.
NFR’s Sentivist dashboards include even more information than TippingPoint, such as a nice real-time timeline showing the number and types of attacks graphically. Sentivist includes almost 20 graphs that can give you differing instantaneous views of your system. Our only complaint with this dashboard is that there’s no easy way to always have multiple graphs showing at the same time. You have to open windows manually, arrange them and redo your work every time you restart the management tool.
Demarc’s Sentarus has a Web-based GUI that looks more like a blog or discussion wiki than an IPS status report, but does manage to pack a lot of valuable status information into a small place. Sentarus also has excellent controls that let you customize the dashboard to include the information you want. We were also impressed to see how Sentarus continues to have some dashboard information part of a constant frame around the entire user interface. This means that some status information is always at your fingertips, even if you weren’t looking directly at the dashboard. The biggest problem we found is that the default dashboard runs queries on the Sentarus back-end database to generate its pretty graphs, and performance of the database with a million events in it was so slow as to make the dashboard an unpleasant stopping place.
We had the same problems evaluating the dashboard in Ambiron TrustWave’s ipAngel as we did with other parts of the product: It looked like they were going in the right direction, but the code was so buggy, and we didn’t get to see it in action.
Top Layer’s IPS 5500 and Fortinet’s FortiGate 3600 have the same fault: They have dashboards, but the dashboards don’t tell you anything particularly useful about the IPS part of the system. It’s like peeling the wax off a piece of cheese and discovering that there’s only wax, and no cheese inside.
For example, the IPS 5500 includes a real-time status page showing blocked and detected events, but the events are sorted by Top Layer ID, an identifier that has absolutely no significance and offers no information.
Task 4: forensics
After an attack, successful or otherwise, the security analyst always wants to know: what happened? What was going on during the attack, who was involved and what else might have happened that I might have missed? Supporting the complex and often-confusing task of dissecting an attack after hours, days or even weeks have gone by is the art of forensics.
More than any other aspect of an IPS, forensics capabilities can strongly differentiate products. However, better forensics do not necessarily make a better IPS. This is one product feature that may be of no interest to you. In smaller organizations, even tuning the IPS may require more resources than are available and forensics are a feature that might never be used.
Forensics features are also stressful to systems and hardware. While building a gigabit-speed IPS is not a simple feat, constructing a system that can store the records for millions of attacks and being able to search, report and analyze those attacks at a speed acceptable to a human analyst can be just as hard. In our testing, some of the management servers handling attack databases were bigger and noisier than the IPS systems themselves.
In our testing, NFR’s Sentivist easily stood above all the other IPS consoles in its forensics capabilities, with TippingPoint’s coming in second, and Top Layer’s a close third.
NFR’s domination in forensics is understandable, because forensics was the original reason for the company’s existence. NFR originally stood for Network Flight Recorder, and its first products were meant to be the network equivalent of an airplane’s flight recorder, giving the critical information about what happened on a network around disastrous events.
NFR’s Sentivist displays this heritage with an easy-to-use forensics interface that even a on which beginner can get up to speed quickly. Sentivist has a clear concept of workflow, with an incident-management system built in that can gather events and alerts for analysis. Alerts can have metadata added to them by the analyst, such as notes or status reports, which can prevent redundant work during an investigation.
More important, Sentivist has the features that you need to get a picture of what happened on your network that might be related to an attack you’re investigating. It’s easy to aggregate events with a click-and-drag interface, so you can quickly spot patterns. And the Sentivist is data-neutral, meaning that whatever field of the event (there are 32 of them available in the analyst interface) is interesting to you can be searched on, aggregated and sorted. That must have been a lot of work behind the scenes, but it’s all flawless and smooth when you sit down at the console. Sentivist also has a fairly nice design for retrieving packet captures around events. While most IPS systems really focus on the individual packet that triggered an event, Sentivist has a more global view of things and keeps packets in a format that makes them easy to slip into a more specialized tool, such as the Ethereal protocol decoder.
If we faulted Sentivist before for being the tool aimed at the full-time security analyst, then that same fault turns into a strength in the area of forensics. Of all the products we looked at, Sentivist is the most capable in forensics and will be the most attractive to someone for whom forensics is a top priority.
The TippingPoint Security Management System also has an impressive forensics interface. Although it doesn’t go quite as deep as NFR into the details, it will be quite adequate for anyone who has occasional use for forensics. TippingPoint’s interface is also well-designed for someone who has to quickly find and tune the IPS for false positives. For example, it’s very easy to click on an event and then edit the filter that caused it, a great way to get rid of false positives.
Ambiron TrustWave’s ipAngel, Demarc’s Sentarus, Fortinet’s FortiGate 3600 and Top Layer’s IPS 5500 all have very simple forensics interfaces. In each you can, essentially, see a long list of events with some sorting capability, but no real tools for the analyst to drill down into in order to analyze and understand the dynamics of an attack. If all the information you need were in a single event, you’d probably be fine with these three, but an attack worth analyzing rarely works that way.
With both ipAngel and Sentarus, we were strongly affected in our forensics by bugs in the version of software that we evaluated. Of course, both vendors told us that all of these issues would be addressed quickly, but because we had been through several rounds of signature or software updates already, we declined the opportunity to test another version.
We also found Fortinet’s FortiGate 3600 to be a poor forensics performer. With an internal 36GB hard drive for logging, the FortiGate is really not well-equipped to be both UTM firewall and IPS management system. Fortinet does offer an add-on product, called Fortilog that has greater log storage capabilities, but did not submit this as part of the IPS test.
Forensics is not just for analyzing attacks. Any IPS is going to throw false positives (an IPS that never has a false positive is an IPS that is not doing its job), and forensics are an important part of finding the false positive and resolving the issue quickly. The ability to tune the signatures quickly during a forensics investigation is critical for any production use of IPS. Without the ability to quickly and easily tune at the filter level, the security analyst would have to completely disable (rather than fine-tune) any signature that throws a false positive on any system, which would quickly diminish the power and value of the IPS. We think that this kind of tuning is so critical that the IPS systems without it, including Demarc’s Sentarus, Fortinet’s FortiGate 3600 and TopLayer’s IPS 5500 should be limited to constrained environments and will not be effective as signature-based IPS systems in large, diverse networks.
Task 5: reporting
The last task we used to analyze usability of IPS management was reporting. Although reporting is often derided by IT professionals as “pretty graphs for managers,” we think that good reporting does bring tremendous value to network managers, security analysts and their bosses.
Reporting should include not only ad hoc reports that can be run in response to specific queries, but also scheduled reports that can be automatically run regularly and published or saved for long-term trend analysis. What goes into an IPS report can be tricky to define. While simply dumping lists of events into a report or a screen display is really a forensics task, aggregating data is a reporting function. Typical report elements might include bandwidth and traffic, rate data (especially useful for tuning a rate-based IPS), common attackers and commonly attacked destination systems, applications and protocols.
There is a complication, though. In an IPS report you’re typically not as interested in the top 10 events (because those are usually either script kiddies or false positives) as you are in the bottom 10 events — the low-and-slow attack or the determined attacker trying to fit under the radar. Representing that kind of information in a report is difficult, because the signal-to-noise ratio is so low. We didn’t find any products that had especially insightful ways to resolve this inherent conflict.
The TippingPoint reporting interface met all of our requirements and had an easy-to-use and intuitive interface. We think that the reporting structure would be more than adequate for most, if not all, IPS users.
Top Layer’s reporting interface edged in just under Tipping Point, because, while Top Layer has many of the same capabilities, there is no automated way to push reports off the system. Below that, we down-rated NFR’s Sentivist, because the reporting isn’t integrated into the management system. Instead, you get dumped into Crystal Reports for a few canned numbers that might keep a manager happy, but won’t really be of use to the analyst. Clearly, NFR doesn’t care about reporting as much as they care about forensics.
Ambiron TrustWave’s ipAngel and Demarc’s Sentarus were crippled by bugs in the reporting function in the software version they submitted for evaluation. Fortinet’s FortiGate 3600 doesn’t have any reporting features, earning it the only failing grade on our usability report card.
Previous: Main IPS performance piece | Next: Why no product stopped Cisco exploit >
