Do you know where your security holes are?

McAfee also did a good job at providing both asset-based and scan-based views of the vulnerability information. For example, MVM was one of two products (QualysGuard VM was the other) that successfully generated a delta report showing the changes in vulnerabilities from one scan to the next.

Where McAfee falls down is in the ability to navigate assets and vulnerabilities in the GUI. Generated reports are easy to produce and read — an advantage over most of the other products. But if you want to view the same information using the Web-based GUI, you're more limited in your ability to navigate through the vulnerabilities and get a good understanding of what your security posture is. We wanted to see reports that had the same level of information whether we read them in a PDF or through the Web-based GUI.

While Qualys and McAfee did very good jobs, we found a lot of promise in SAINT and eEye. Both products have added free report writers to their solid scanners. SAINTwriter and Insight are a way for these two vendors to move from older scan-only products into the enterprise world. However, our testing shows that both have some distinct drawbacks.

SAINT includes 16 pre-defined report types in SAINTwriter. If those 16 types (including asset-based and vulnerability-based reports) meet your needs, you'll be much happier with SAINT than if you need to use its custom interface to define reports.

Compared, for example, to the design of eEye's report writing system, adding reports to SAINT is painful and difficult. SAINT also has some distinct limitations. For example, dynamic queries ("show me all the highest criticality vulnerabilities" or "show me all the Microsoft patches") can't be created in the report definition tool.

SAINT also makes it difficult to navigate through scan results in the GUI. Finding a single system on the network requires considerable navigation, and if your network has hundreds of systems in the vulnerability scanner database, SAINT makes it nearly impossible to be in control. On the other hand, once you find the system you want, there are good tools for both exploring and managing vulnerabilities, such as a one-click "ignore this vulnerability for this system" and an equally nice "ignore this vulnerability for all systems" option.

Although eEye's developers have built a very attractive Shockwave Flash-based GUI into Retina for reporting and vulnerability navigation, we found severe functionality issues that made using the product frustrating and got in the way of effectively managing vulnerabilities.

The near-absence of any documentation, especially on critical functionality of the product, didn't help. Retina is a very new product, about a year old, built on top of eEye's venerable and well-respected scanner. During the three months we were testing, we saw one upgrade of Retina, and eEye released another just before we went to press to address two of our major complaints: the inability to ignore vulnerabilities for some systems, and the inability to run delta reports.

Compared to all the other tools we tested, defining new reports is a dream in Retina. The Retina GUI is, in fact, all report-driven: You tell it what it needs to report on, and it goes out and scans to get the data (or re-uses old data if you want a different format report on the same old data). Once you define a report and run it, though, good luck: All the reports show up with the generic report title — you can't title a specific run, such as "east coast" or "west coast," without creating a whole new report template. That makes finding your results a little difficult and you have to click through each report to see if you can figure out which is which.

Overall, Retina's reporting and analysis capabilities have potential to leap ahead and become a great tool. Today, though, Retina's reporting toolkit doesn't offer the usability or functionality that an enterprise needs for large-scale vulnerability assessment. However, that story could change very quickly.

We thought that the reporting functionality in Critical Watch and Lumension was not up to the standards of the other products.

Critical Watch's interface fails the most basic human factors design, with features like triple nested scrollbars, lack of customization of the view screen and poor integration with its own policy management system. For example, when you see a vulnerability and want to filter it out of future reports (as a false positive, for example), the GUI requires eight or more clicks, and when you're done, you've lost your place in the report.

While there is general trend information in FusionVM, and other well-designed reports, including variance from expectations, simple delta reporting isn't possible. With FusionVM, we found other reporting interoperability issues, such as PDF reports that are incompatible with some readers (possibly because they are encrypted) and Excel reports that won't import into Excel.

We were also frustrated with Lumension's reporting features. Automatic generation of reports on a schedule isn't possible, and our short-term trending report was a 2,800 page listing on two scans less than 24 hours apart. Because of Lumension's scan-based focus, we found that trying to look at data on anything other than the entire scan (such as a subset of systems scanned that are of greater interest) was essentially impossible without re-scanning or re-reporting.

Lumension has a simpler reporting and analysis interface than the other products. This makes it easy to get started and find things, but once you hit the functionality wall, there's not a lot of room for creative configurations.

Manageability and workflow

With hundreds or thousands of systems being scanned, management of configuration is destined to grow complicated over times. Devices will have "ignore" rules (a particular vulnerability shouldn't be reported, for example), and policy exceptions and settings will get more and more customized.

The goal of network managers is to have their weekly vulnerability report as short as possible, highlighting only the important and new issues that have to be considered, and the only way to do that is to make the vulnerability analyzer a very network-aware system. The less time spent micro-managing the vulnerability analyzer, the more time is available to actually fix vulnerabilities, so ease of use and a well thought-out GUI are important. One of our core evaluation criteria was how well these products would fit into a continuous cycle of security posture management, rather than a one-time scan.

We think that McAfee offers the best level of manageability of any of the products we tested. A good example is credential configuration. Credentials are a weak part of vulnerability scanning — you have to give the scanner the right to log onto each system, but having those usernames and passwords floating around is a security threat. In addition, for Windows, the credentials can't be just any user, as some elevated privileges are required to fully evaluate patch levels and registry settings.

McAfee has the best credential management system of any of the products. Credentials are managed and stored separately, making it easy to bring them in, as needed, for any scans. Compare, for example, to eEye, where credentials are attached to particular scanning jobs, rather than assets, which means that they may have to be entered over and over again as different jobs are run across the network.

Other parts of McAfee's management system stood out as helping the network manager get a handle on scanning and scan policies quickly and easily. For example, vulnerabilities are grouped in an intuitive way by category and operating system. Pick the appropriate groupings and apply to a scan job, and you've trimmed things down to a safe set for your network. This is an important capability. Any vulnerability scanner vendor that says they always detect operating systems and only do safe scans isn't being honest or doesn't know their product very well. McAfee isn't the only vendor to bring this capability, but some slice things into absurd segments, such as Lumension, which made us differentiate between Windows 2003, Windows 2008 and Windows 2008 SP2.

In other areas, we found a similar feature, but better implementation in some products. For example, delegated management is an important feature, as it allows the network manager to break up their network, and focus reporting on the individuals responsible in each area.

Everyone had it, but Qualys and CriticalWatch had the best-designed delegated management, effectively separating scanning and reporting functions. This makes sure that two individuals looking at the same set of results could see two different views depending on their responsibilities. We had the most problems with eEye's delegated management, which is great on the scanning side but doesn't extend to the reporting side. In fact, Retina, incomprehensibly, has two completely separate authentication systems, one for reporting and one for everything else.

A very basic task for vulnerability analyzers is handling vulnerabilities: assigning them to someone to fix, marking them as ignored, or even just putting them off for a few weeks until patching can occur. Together, we categorized all of this as "workflow."

We had mixed success in looking at Critical Watch. The product has an integrated ticketing system called Remediation Manager, but it wouldn't work with Firefox, Safari or Internet Explorer browsers, which made it a bit of a non-starter for our testing. On the other hand, it has a fantastic Filter Manager, which is used to selectively mask vulnerabilities at several levels very easily.

EEye took a different route by partnering with other security vendors, including trouble ticket products, as well as Security Information Manager (SIM) products. Getting tickets out of Retina is easy, if you have another product to manage them. One of Retina's strengths is that it integrates very tightly with eEye's end-point security product, Blink. The combination of Retina and Blink together form a more holistic view of end-point security by integrating protection policies (such as end-point firewall configurations) and vulnerability scanning. This offers the capability to mitigate known attacks before patches are available or installed. Retina CS also gets the prize for coolest-looking management system, even if some of the new AJAX interface objects are decidedly non-functional and poorly thought-out.

Lumension Scan, like Retina, works best when integrated into the vendor's own endpoint tools. Lumension's patch management fits cleanly with Lumension Scan to help drive a more integrated solution all the way to the server or workstation. Without patch management, we found that Lumension Scan had a particularly primitive workflow. To ignore a discovered vulnerability, you'd have to write down the information from a report and then go and reconfigure a job not to include that vulnerability. On the other hand, Lumension Scan had some great management tools. For example, once a vulnerability was considered fixed, you could re-test just that vulnerability on just that one system with a single click, which was a great convenience. And, if Lumension Scan identified a vulnerability in some configuration parameter, the information on how and why to fix it was built directly into the management interface, saving time and energy.

Workflow in McAfee was unnecessarily complicated. Vulnerabilities can flow directly into an integrated trouble ticketing system (or externally via SMTP or SNMP) and be assigned to users based on various rules. At that point, it's easy to ignore a discovered vulnerability or mark it as fixed or a false positive. But we found navigation inconsistent when thinking about workflows. In some cases, for example, you can tell it to ignore something, but in other places that might be just as appropriate, you can't.

That inconsistency was true with QualysGuard. We discovered that some views within the GUI let you change system behavior while others don't, often without rhyme or reason. Overall QualysGuard VM suffers in its management interface from an aging design that spreads too much information across too many menus and uses the same terminology in too many different ways. Once you're past the learning curve, it isn't hard to get things accomplished, but there are confusing bits here and there which never quite make sense.

And if any product needs a GUI and workflow redesign, it's SAINT. However, we learned quickly not to judge a product by its GUI. Even if the design of the screens screams "1997," the product has a great deal of power hidden beneath. The integrated trouble-ticketing system, for example, is surprisingly easy to use and has a well-designed rule base that can be used to automatically assign trouble tickets to the appropriate group.

Where's my favorite vulnerability analysis vendor?

For this test, we wrote an extensive evaluation criteria document and sent invites to 13 vendors. IBM (through its ISS acquisition) declined, telling us that it was going to a cloud-based solution to replace its existing product. NCircle and Trustwave both declined, telling us that vulnerability assessment was only a piece of their products and they didn't fit in very well to our test. StillSecure and Rapid7 both acknowledged our invitation, but never replied. Beyond Security wouldn't reply or acknowledge our invitation. Tenable, keepers of Nessus, the most popular semi-open-source scanner in the marketplace said that it wanted to participate, but never actually sent us product to test.

Snyder is a senior partner at Opus One in Tucson, Ariz. He can be reached at Joel.Snyder@opus1.com.