Vulnerability management tools in a nutshell

We tested six market-leading products and evaluated each for their vulnerability scanner results, reporting features, product manageability, workflow tools, and interoperability with other enterprise products. Here are capsule reviews of each product.

Critical Watch FusionVM Enterprise

We tested FusionVM in its software-as-a-service configuration, giving us a portal-based vulnerability analyzer with off-site and on-site scanner capability. (Critical Watch offers other packagings which are entirely on-site if needed.)

Do you know where your security holes are?

We found it nice to be able to quickly deploy scanners in virtual machines. Having the flexibility to scan from the inside or the outside also gives additional benefits. With strong feature sets focusing on delegated management and compliance, FusionVM has a clear emphasis on the compliance marketplace. We found the reporting to be a strong feature, and the built-in Web vulnerability testing features will be interesting to anyone who fears bugs in their externally facing Web sites.

FusionVM also has a direct link to TippingPoint IPS products, offering the ability to optimize IPS configurations based on real detected vulnerabilities and systems.

In comparing FusionVM to other vulnerability analyzers, we found much to like and some features that didn't thrill us. The documentation is out-of-date and, in many of the places we went looking, poorly executed. The Web-based GUI, so critical to management of FusionVM, didn't work very well in our environment, with some features such as remediation and workflow blocked almost completely, while other usability flaws got in our way for common operations. If these features can be remedied, FusionVM will be a strong competitor, but there is work to be done at Critical Watch.

eEye Retina CS

Retina CS (Compliance + Security) is a reporting and compliance toolkit and GUI that sits on top of the venerable and well-respected Retina Network Security Scanner. CS is a relatively new product (about a year old) and shows some rough edges, scanner and GUI bugs, and design flaws.

Compliance considerations

But the product is under active development. We saw one upgrade of Retina CS during our test, and one slated to appear after we were done (with a feature that we really wanted, exclusion listing).

Retina CS stood out for its easy-to-define report formats. What we found missing, though, was solid integration between the scanner reporting engine and the database of scan information, requiring not only separate GUIs but even separate authentication systems. The strong set of third-party partners reflects the maturity of the company and its long-time presence in this marketplace. Retina CS is a favorite underdog, with some great ideas and technology that need further refinement and a good dose of bug fixes. When Retina CS does finally become enterprise-ready, it'll be a great tool for network managers who have systems, Web site, and compliance concerns.

Lumension Scan

Lumension Scan is a very nicely constrained product that doesn't try to reach beyond its base functionality and capabilities. Our extensive enterprise feature set was a bit of a challenge for Lumension, and so it didn't rate so well in our scorecard, but this doesn't mean it isn't a fine product — it just doesn't have the bells and whistles we were looking for in this test.

Lumension has made a nice trade-off in the product by limiting its complexity, features and cost all at the same time. This gives network managers a nice choice for basic vulnerability analysis and basic reporting, at a cost which may be more affordable than the feature-rich alternatives. Network managers need to be aware of where Lumension Scan hits the wall, such as in managing long-term results. However, for checking on patch status, verifying basic compliance policies, and rolling out reports quickly, Lumension Scan is a solid performer.

McAfee Vulnerability Manager

McAfee Vulnerability Manager (MVM) is delivered as either software or a pre-built appliance. In either case, we found MVM to be one of the most mature products in our testing. MVM's roots are in the 2004 acquisition of Foundstone, and the MVM product still has some Foundstone branding in the core. We found MVM to be built solidly and reliable from top to bottom. Some aspects, such as the system for storing scanning credentials, and outstanding deployment documentation, should be lessons for all the other vendors in this space.

Our biggest complaints about MVM are in the aggressiveness of the scanner, as MVM was responsible for causing more problems on our network than any other. Some aspects of the product show the difficulty of adding a new GUI on top of an older scanner (something we saw in Retina CS as well), and McAfee has not been extremely successful in convincing us of the advantages of their real-time threat alerting system.

McAfee's deep involvement in OVAL, the emerging standard for vulnerability analysis, definitely pays off in MVM as features such as compliance scanning are drastically simplified when moving between different regulatory regimes.

QualysGuard Enterprise Suite

QualysGuard is a software-as-a-service vulnerability analysis system with the option for both on-site (hardware-based) and cloud-based scanners. QualysGuard has an outstanding product with an excellent feature set and well-thought-out reporting from top to bottom. While the SaaS model has some weaknesses, such as the inability to automatically run reports easily, QualysGuard emerged as a clear leader in our testing, both from an accuracy and a usability point of view. If you are looking for a product to help with long-term management of security posture, QualysGuard will fit cleanly into your workflow almost without any documentation or training.

We did find some nits to pick with QualysGuard, particularly in the amount of effort it takes to divide up systems and responsibilities. But these problems pale compared to our overall verdict of QualysGuard as a top solution to the problem of long-term management of vulnerabilities in enterprises. Having a SaaS approach hasn't resulted in many compromises, and the entire product makes it easy to get good, actionable, data and move to solving problems on your network and improving on your overall security posture.

SAINTscanner

SAINT is the do-it-yourself version of the vulnerability scanner. SAINT has the deepest roots of any scanner, being the direct descendent of SATAN, released back in 1995. SAINT Corporation does offer SaaS-based and appliance-based versions of the SAINTscanner, but even then the product retains its box-of-tools style. SAINT goes further in the penetration testing business than most of the vulnerability analyzer products, with add-ons such as SAINTexploit that can be used to prove the presence of a vulnerability by launching an exploit.

SAINTScanner's weak point is the management interface, which is amateurish at best. We found that the interface was especially deceptive — it looks and feels so primitive that you don't really understand the power underlying it. We found that looking at the reports and output from SAINT gave us a stronger idea of how good of a product was running under the covers. Network managers who need to think outside of the prepackaged box will find that SAINT offers more possibilities to get deep into the product than any other commercial offering.