Do you know where your security holes are?

We all worry that there’s some lurking security problem in our servers. We do what we can, patching, following best practices, keeping up-to-date with training and news. But wouldn’t it be great to have an automated tool to check our work? That’s the promise of vulnerability analyzers: products that detect problems in configuration, applications, and patches.

Used correctly, a vulnerability analyzer can help you stay on top of hundreds or thousands of servers, network devices, and embedded systems. You’ll know where to focus your efforts for security remediation, and you’ll know that you have a system in place to keep little things from slipping through the cracks and becoming big things.

However, used incorrectly, these analyzers can generate thousands of pages of confusing information, frustrate security and network managers, and end up causing more problems than they solve.

We evaluated six market-leading products for their vulnerability scanner results, reporting features, product manageability, workflow tools, and interoperability with other enterprise products.

Two products stood out: SaaS-based QualysGuard VM, and McAfee’s Vulnerability Manager, a software or appliance-based product.

SAINTmanager product line came in third, buoyed by a powerful scanner, but burdened by a weak GUI. Our favorite challenger, eEye Retina CS, paired a strong scanner with a newly-minted GUI. But we found a number of bugs and design flaws that need to be fixed before the product is ready for enterprise deployment. Retina is a relatively new product that is under active development. During the three months we were testing, we saw one upgrade of Retina, and eEye released another just before we went to press.

Compliance considerations

Critical Watch’s FusionVM product, another SaaS-based offering, has some great ideas in it, but the execution is lacking. Lumension Scan, a product with a more limited scope, did a good job at what it was designed for, but didn’t have the enterprise focus we were seeking.

Scanning for stuff

All vulnerability analyzers have a common core: a scanner that finds vulnerabilities. If you were doing a penetration test on a network, the scanner would be all that you need.

In some products, notably eEye’s Retina CS and SAINT Corporation’s SAINTscanner/SAINTmanager/SAINTwriter, the scanner is a standalone entity that you could run without the reporting and management tools. (See Web scanning as an option.)

In others, such as QualysGuard VM and Critical Watch’s FusionVM, the scanner is inseparable from the other pieces . If you’re a security consultant who wants to just perform scans, eEye and SAINT will fit your needs best.

To get an idea of how well the scanners worked, we scanned three production networks at three companies, plus a specially-constructed test lab network. On the test lab network, we deliberately let four servers fall behind in their patches by two months: two Windows systems (Windows 2003 and 2008), a Linux system, and an OS X server. (Read how we conducted our tests.)

Then we turned on the vulnerability analyzers and evaluated the results.

First, a word of caution: vulnerability scanners can and will cause instability in your network. SAINT and Critical Watch did real damage on our network, managing to lock up one of our production Unix servers, and causing SAN hiccups that interrupted service to several clients.

Virtually all of the products caused our APC UPSs to reboot, which (fortunately) didn’t affect anything but the management interfaces. So, be careful what IP addresses you scan and how the scans are run.

You’d think our networks wouldn’t have gotten too far out of date in only two months, but these scanners had a lot to say. The winner by weight was eEye, which dumped a 180-page report on our desk, although McAfee won by count, telling us 537 different things about those four systems, 380 of which weren’t specific vulnerabilities, but only informational items. Still, that left 84 critical vulnerabilities that McAfee wanted us to fix.

An obvious conclusion was that some vendors weren’t doing a good job of data reduction. Yes, it’s true that Adobe patch APSB10-14 covers 28 distinct vulnerabilities, but they’re all fixed by a single patch, and treating that as 28 separate incidents — as McAfee does — simply encourages confusion.

Critical Watch had a similar issue, delivering all of the pieces of some of Microsoft‘s critical patches as separate elements, even though the remediation task was the same: install MS11-012 to fix five separately reported vulnerabilities. The pedantic security wonk might insist on knowing about each separate issue, but that’s what drill-down reports are for. By default, this information should be combined into a more digestible format.

Some results were very cut-and-dried. For example, on our test of Windows systems, we had a list of security updates and patches that Microsoft Update gave us and we expected to see each of those patches in the list of vulnerabilities for each system. With McAfee, Qualys, and eEye, we found everything in our checklist in their scan results. Critical Watch, Lumension, and SAINT didn’t catch everything.

If that checklist of known missing security patches was our entire test, it would have been easy to rank products. But each of the scanners had lots to tell us, and figuring out whether or not the information was relevant or worthwhile was tricky.

For example, Lumension told us that we were running an out-of-date version of SecureCRT (true) on our Windows systems and rated the vulnerability as “high” (probably over-kill). No other product we tested picked up on this. Does this mean that Lumension is better than the other scanners, which didn’t catch this problem?

Well, yes, except that eEye found a circa-1999 protection setting on an obscure registry key that could be used by privileged users to further escalate their privileges during system boot. Does that make eEye better than the other scanners, which didn’t identify the registry problem? You can go in this circle forever, as each product called out issues, mostly minor ones, that the others didn’t.

False positives

We also looked at false positives, places where the scanners reported a vulnerability where none existed. This is a sensitive area for security purists. Some scanners simply treated the existence of a particular file as making a system vulnerable. For example, Lumension marked the Unix kernel on our Linux test system as out-of-date, probably because it saw an installed Red Hat Package Manager with the old kernel. Except that we weren’t running that kernel, so the system wasn’t vulnerable. But the kernel was out there, spinning around on disk.

Of course, not all false positives were so difficult to judge. For example, CriticalWatch identified our Mac OS X system as missing a Microsoft Windows patch, and eEye wanted patches for VMware and HyperV vulnerabilities on systems that didn’t have either installed. We identified false positives in every scanner but Qualys.

We had other challenges in comparing scanners. The CVE (Common Vulnerabilities and Exposures) database supported by the U.S. Department of Homeland Security is the closest thing we have to a common naming standard. We would have expected vendors to make sure that their vulnerabilities were properly matched to CVE database entries, but we found errors and omissions in most of the scanners when trying to compare results. For example, we thought McAfee had identified a problem no one else had caught with CVE-2010-3886, until we found it hidden in eEye as Retina Audit 13156, without the CVE number.

We also looked at two other important criteria: reporting on evidence of the vulnerability, and cross-platform support.

Where’s the evidence?

We found evidence, or the lack of it — a big differentiator. When a scanner reports a vulnerability, it is critical to have supporting evidence readily available. For example, one of the scanners reported a vulnerability of “user never logged in” on our Windows servers, without telling us which user it was. How useful is that? (Answer: not very.)

McAfee, Qualys and SAINT gave us great supporting evidence, with Critical Watch providing some evidence but not as much as we wanted. EEye and Lumension were deficient in providing evidence, although eEye did hide some of the evidence in reports that were invisible in the Web interface.

Scanning different operating system platforms was another differentiator which may not be important to every network manager. All products supported Windows, but we found varying levels of commitment to non-Windows systems, such as Mac and Unix, as well as infrastructure devices such as switches, databases and routers. SAINT, and to a lesser extent, eEye, had a stronger focus on Unix systems than the other products.

Overall, we felt that Qualys and SAINT had the strongest core scanners, with McAfee nearly at the same level. Lumension, eEye and Critical Watch had higher numbers of false positives and false negatives, as well as poor evidence presentation when documenting vulnerabilities.

Reporting

Vulnerability information hidden inside a product doesn’t do network managers any good. We expect a vulnerability assessment product to be able to report on the information it finds in a way that maximizes understanding and minimizes wasted time.

Network managers will probably want to use a GUI for their reports, because they’ll be diving down into details, looking at individual systems and vulnerabilities and focusing on tasks such as remediation and patching.

Auditors and managers may want printed reports that summarize information, giving a big picture view of where the enterprise is at risk and where they need to focus attention. We looked at all types of output under the general category of “reporting,” to evaluate understandability, transparency, configurability and usefulness.

We discovered that there are really two types of vulnerability analyzers out there: scan-based and asset-based. A scan-based analyzer is one where all the reports and data are focused on scanning jobs. In other words, the analyzer runs a scan job, and then you can report on that job.

Since the vulnerability scanner runs by looking at a list of targets, running some selected scans and generating output, it’s very natural to build a scan-based vulnerability analyzer. You just take the scanner that you worked so hard on and stick a fancy GUI on top of that. The scanner may track information such as trends, but it’s not maintaining a detailed history on each system being scanned.

The alternative to a scan-based analyzer is an asset-based analyzer. (An “asset” can be a server, workstation, router or whatnot.) Here the focus is not so much on scanning jobs, but on information that is collected about assets. Over time, the scanner picks up bits and pieces of information about assets on your networks, building up a picture of vulnerabilities, configurations, and even patching history. Some of the vendors have started to call this “scan once, report many” in their literature.

If you’re running a single scan for a PCI audit or a penetration test, you’ll be happy with scan-based vulnerability analyzers. However, if you’re trying to incorporate vulnerability management as part of a continuous process, aimed at understanding the security posture across an enterprise network over time, you’ll want an asset-based analyzer.

With this approach, you might take a long time to set up your scans, but you would expect to run them regularly and repeatedly, and contribute to a “body of knowledge” about your network.

An asset-based analyzer can always behave as a scan-based analyzer by simply reporting on the latest scan, and each of the products we tested did that perfectly. However, we had varying degrees of success getting products to zoom out of the scans and give us a big picture in their reports across multiple scans, over time.

Qualys came closest to our ideal of what perfect vulnerability analyzer reporting would act like, with McAfee nearly at the same level. Both let you generate easy to read and navigable reports, both scan-based and asset-based. The reporting interfaces in both are intuitive, and we strongly preferred the idea that reporting could be very detached from scanning.

However, neither product was perfect. Qualys wouldn’t let us run automated reports. That’s a side-effect of its strong SaaS security model. Because Qualys is holding extremely sensitive information in its databases about your network and systems, including credentials you might have provided for scanning purposes, Qualys takes security very seriously.

In this case, maybe a little too seriously. Its encryption model keeps anyone from looking at your enterprise data unless you’re actually logged into the Web-based user interface. Unfortunately, “anyone” in this case also includes Qualys’ report writer. It doesn’t run when you’re not logged in, which means that getting Qualys to generate reports and send them to you on a regular basis is about 100 times harder than it should be. You can do it, if you insist.

The Qualys team gave us information on the QualysGuard API, a tool that would let someone remotely launch reports, perhaps out of a batch job. However, that’s not really what we were looking for. Qualys is proud of how seriously it takes the security of sensitive data, but this is one case where it needs to figure out a simpler way to perform a simple task.

McAfee also did a good job at providing both asset-based and scan-based views of the vulnerability information. For example, MVM was one of two products (QualysGuard VM was the other) that successfully generated a delta report showing the changes in vulnerabilities from one scan to the next.

Where McAfee falls down is in the ability to navigate assets and vulnerabilities in the GUI. Generated reports are easy to produce and read — an advantage over most of the other products. But if you want to view the same information using the Web-based GUI, you’re more limited in your ability to navigate through the vulnerabilities and get a good understanding of what your security posture is. We wanted to see reports that had the same level of information whether we read them in a PDF or through the Web-based GUI.

While Qualys and McAfee did very good jobs, we found a lot of promise in SAINT and eEye. Both products have added free report writers to their solid scanners. SAINTwriter and Insight are a way for these two vendors to move from older scan-only products into the enterprise world. However, our testing shows that both have some distinct drawbacks.

SAINT includes 16 pre-defined report types in SAINTwriter. If those 16 types (including asset-based and vulnerability-based reports) meet your needs, you’ll be much happier with SAINT than if you need to use its custom interface to define reports.

Compared, for example, to the design of eEye’s report writing system, adding reports to SAINT is painful and difficult. SAINT also has some distinct limitations. For example, dynamic queries (“show me all the highest criticality vulnerabilities” or “show me all the Microsoft patches”) can’t be created in the report definition tool.

SAINT also makes it difficult to navigate through scan results in the GUI. Finding a single system on the network requires considerable navigation, and if your network has hundreds of systems in the vulnerability scanner database, SAINT makes it nearly impossible to be in control. On the other hand, once you find the system you want, there are good tools for both exploring and managing vulnerabilities, such as a one-click “ignore this vulnerability for this system” and an equally nice “ignore this vulnerability for all systems” option.

Although eEye’s developers have built a very attractive Shockwave Flash-based GUI into Retina for reporting and vulnerability navigation, we found severe functionality issues that made using the product frustrating and got in the way of effectively managing vulnerabilities.

The near-absence of any documentation, especially on critical functionality of the product, didn’t help. Retina is a very new product, about a year old, built on top of eEye’s venerable and well-respected scanner. During the three months we were testing, we saw one upgrade of Retina, and eEye released another just before we went to press to address two of our major complaints: the inability to ignore vulnerabilities for some systems, and the inability to run delta reports.

Compared to all the other tools we tested, defining new reports is a dream in Retina. The Retina GUI is, in fact, all report-driven: You tell it what it needs to report on, and it goes out and scans to get the data (or re-uses old data if you want a different format report on the same old data). Once you define a report and run it, though, good luck: All the reports show up with the generic report title — you can’t title a specific run, such as “east coast” or “west coast,” without creating a whole new report template. That makes finding your results a little difficult and you have to click through each report to see if you can figure out which is which.

Overall, Retina’s reporting and analysis capabilities have potential to leap ahead and become a great tool. Today, though, Retina’s reporting toolkit doesn’t offer the usability or functionality that an enterprise needs for large-scale vulnerability assessment. However, that story could change very quickly.

We thought that the reporting functionality in Critical Watch and Lumension was not up to the standards of the other products.

Critical Watch’s interface fails the most basic human factors design, with features like triple nested scrollbars, lack of customization of the view screen and poor integration with its own policy management system. For example, when you see a vulnerability and want to filter it out of future reports (as a false positive, for example), the GUI requires eight or more clicks, and when you’re done, you’ve lost your place in the report.

While there is general trend information in FusionVM, and other well-designed reports, including variance from expectations, simple delta reporting isn’t possible. With FusionVM, we found other reporting interoperability issues, such as PDF reports that are incompatible with some readers (possibly because they are encrypted) and Excel reports that won’t import into Excel.

We were also frustrated with Lumension’s reporting features. Automatic generation of reports on a schedule isn’t possible, and our short-term trending report was a 2,800 page listing on two scans less than 24 hours apart. Because of Lumension’s scan-based focus, we found that trying to look at data on anything other than the entire scan (such as a subset of systems scanned that are of greater interest) was essentially impossible without re-scanning or re-reporting.

Lumension has a simpler reporting and analysis interface than the other products. This makes it easy to get started and find things, but once you hit the functionality wall, there’s not a lot of room for creative configurations.

Manageability and workflow

With hundreds or thousands of systems being scanned, management of configuration is destined to grow complicated over times. Devices will have “ignore” rules (a particular vulnerability shouldn’t be reported, for example), and policy exceptions and settings will get more and more customized.

The goal of network managers is to have their weekly vulnerability report as short as possible, highlighting only the important and new issues that have to be considered, and the only way to do that is to make the vulnerability analyzer a very network-aware system. The less time spent micro-managing the vulnerability analyzer, the more time is available to actually fix vulnerabilities, so ease of use and a well thought-out GUI are important. One of our core evaluation criteria was how well these products would fit into a continuous cycle of security posture management, rather than a one-time scan.

We think that McAfee offers the best level of manageability of any of the products we tested. A good example is credential configuration. Credentials are a weak part of vulnerability scanning — you have to give the scanner the right to log onto each system, but having those usernames and passwords floating around is a security threat. In addition, for Windows, the credentials can’t be just any user, as some elevated privileges are required to fully evaluate patch levels and registry settings.

McAfee has the best credential management system of any of the products. Credentials are managed and stored separately, making it easy to bring them in, as needed, for any scans. Compare, for example, to eEye, where credentials are attached to particular scanning jobs, rather than assets, which means that they may have to be entered over and over again as different jobs are run across the network.

Other parts of McAfee’s management system stood out as helping the network manager get a handle on scanning and scan policies quickly and easily. For example, vulnerabilities are grouped in an intuitive way by category and operating system. Pick the appropriate groupings and apply to a scan job, and you’ve trimmed things down to a safe set for your network. This is an important capability. Any vulnerability scanner vendor that says they always detect operating systems and only do safe scans isn’t being honest or doesn’t know their product very well. McAfee isn’t the only vendor to bring this capability, but some slice things into absurd segments, such as Lumension, which made us differentiate between Windows 2003, Windows 2008 and Windows 2008 SP2.

In other areas, we found a similar feature, but better implementation in some products. For example, delegated management is an important feature, as it allows the network manager to break up their network, and focus reporting on the individuals responsible in each area.

Everyone had it, but Qualys and CriticalWatch had the best-designed delegated management, effectively separating scanning and reporting functions. This makes sure that two individuals looking at the same set of results could see two different views depending on their responsibilities. We had the most problems with eEye’s delegated management, which is great on the scanning side but doesn’t extend to the reporting side. In fact, Retina, incomprehensibly, has two completely separate authentication systems, one for reporting and one for everything else.

A very basic task for vulnerability analyzers is handling vulnerabilities: assigning them to someone to fix, marking them as ignored, or even just putting them off for a few weeks until patching can occur. Together, we categorized all of this as “workflow.”

We had mixed success in looking at Critical Watch. The product has an integrated ticketing system called Remediation Manager, but it wouldn’t work with Firefox, Safari or Internet Explorer browsers, which made it a bit of a non-starter for our testing. On the other hand, it has a fantastic Filter Manager, which is used to selectively mask vulnerabilities at several levels very easily.

EEye took a different route by partnering with other security vendors, including trouble ticket products, as well as Security Information Manager (SIM) products. Getting tickets out of Retina is easy, if you have another product to manage them. One of Retina’s strengths is that it integrates very tightly with eEye’s end-point security product, Blink. The combination of Retina and Blink together form a more holistic view of end-point security by integrating protection policies (such as end-point firewall configurations) and vulnerability scanning. This offers the capability to mitigate known attacks before patches are available or installed. Retina CS also gets the prize for coolest-looking management system, even if some of the new AJAX interface objects are decidedly non-functional and poorly thought-out.

Lumension Scan, like Retina, works best when integrated into the vendor’s own endpoint tools. Lumension’s patch management fits cleanly with Lumension Scan to help drive a more integrated solution all the way to the server or workstation. Without patch management, we found that Lumension Scan had a particularly primitive workflow. To ignore a discovered vulnerability, you’d have to write down the information from a report and then go and reconfigure a job not to include that vulnerability. On the other hand, Lumension Scan had some great management tools. For example, once a vulnerability was considered fixed, you could re-test just that vulnerability on just that one system with a single click, which was a great convenience. And, if Lumension Scan identified a vulnerability in some configuration parameter, the information on how and why to fix it was built directly into the management interface, saving time and energy.

Workflow in McAfee was unnecessarily complicated. Vulnerabilities can flow directly into an integrated trouble ticketing system (or externally via SMTP or SNMP) and be assigned to users based on various rules. At that point, it’s easy to ignore a discovered vulnerability or mark it as fixed or a false positive. But we found navigation inconsistent when thinking about workflows. In some cases, for example, you can tell it to ignore something, but in other places that might be just as appropriate, you can’t.

That inconsistency was true with QualysGuard. We discovered that some views within the GUI let you change system behavior while others don’t, often without rhyme or reason. Overall QualysGuard VM suffers in its management interface from an aging design that spreads too much information across too many menus and uses the same terminology in too many different ways. Once you’re past the learning curve, it isn’t hard to get things accomplished, but there are confusing bits here and there which never quite make sense.

And if any product needs a GUI and workflow redesign, it’s SAINT. However, we learned quickly not to judge a product by its GUI. Even if the design of the screens screams “1997,” the product has a great deal of power hidden beneath. The integrated trouble-ticketing system, for example, is surprisingly easy to use and has a well-designed rule base that can be used to automatically assign trouble tickets to the appropriate group.

Where’s my favorite vulnerability analysis vendor?

For this test, we wrote an extensive evaluation criteria document and sent invites to 13 vendors. IBM (through its ISS acquisition) declined, telling us that it was going to a cloud-based solution to replace its existing product. NCircle and Trustwave both declined, telling us that vulnerability assessment was only a piece of their products and they didn’t fit in very well to our test. StillSecure and Rapid7 both acknowledged our invitation, but never replied. Beyond Security wouldn’t reply or acknowledge our invitation. Tenable, keepers of Nessus, the most popular semi-open-source scanner in the marketplace said that it wanted to participate, but never actually sent us product to test.

Snyder is a senior partner at Opus One in Tucson, Ariz. He can be reached at Joel.Snyder@opus1.com.