Skip Links

Do you know where your security holes are?

Qualys and McAfee lead the way in six-vendor test of automated tools that scan and report on vulnerabilities

By , Network World
June 20, 2011 12:08 AM ET

Network World - We all worry that there's some lurking security problem in our servers. We do what we can, patching, following best practices, keeping up-to-date with training and news. But wouldn't it be great to have an automated tool to check our work? That's the promise of vulnerability analyzers: products that detect problems in configuration, applications, and patches.

Used correctly, a vulnerability analyzer can help you stay on top of hundreds or thousands of servers, network devices, and embedded systems. You'll know where to focus your efforts for security remediation, and you'll know that you have a system in place to keep little things from slipping through the cracks and becoming big things.

However, used incorrectly, these analyzers can generate thousands of pages of confusing information, frustrate security and network managers, and end up causing more problems than they solve.

We evaluated six market-leading products for their vulnerability scanner results, reporting features, product manageability, workflow tools, and interoperability with other enterprise products.

Two products stood out: SaaS-based QualysGuard VM, and McAfee's Vulnerability Manager, a software or appliance-based product.

SAINTmanager product line came in third, buoyed by a powerful scanner, but burdened by a weak GUI. Our favorite challenger, eEye Retina CS, paired a strong scanner with a newly-minted GUI. But we found a number of bugs and design flaws that need to be fixed before the product is ready for enterprise deployment. Retina is a relatively new product that is under active development. During the three months we were testing, we saw one upgrade of Retina, and eEye released another just before we went to press.

Compliance considerations

Critical Watch's FusionVM product, another SaaS-based offering, has some great ideas in it, but the execution is lacking. Lumension Scan, a product with a more limited scope, did a good job at what it was designed for, but didn't have the enterprise focus we were seeking.

Scanning for stuff

All vulnerability analyzers have a common core: a scanner that finds vulnerabilities. If you were doing a penetration test on a network, the scanner would be all that you need.

In some products, notably eEye's Retina CS and SAINT Corporation's SAINTscanner/SAINTmanager/SAINTwriter, the scanner is a standalone entity that you could run without the reporting and management tools. (See Web scanning as an option.)

In others, such as QualysGuard VM and Critical Watch's FusionVM, the scanner is inseparable from the other pieces . If you're a security consultant who wants to just perform scans, eEye and SAINT will fit your needs best.

To get an idea of how well the scanners worked, we scanned three production networks at three companies, plus a specially-constructed test lab network. On the test lab network, we deliberately let four servers fall behind in their patches by two months: two Windows systems (Windows 2003 and 2008), a Linux system, and an OS X server. (Read how we conducted our tests.)

Then we turned on the vulnerability analyzers and evaluated the results.

First, a word of caution: vulnerability scanners can and will cause instability in your network. SAINT and Critical Watch did real damage on our network, managing to lock up one of our production Unix servers, and causing SAN hiccups that interrupted service to several clients.

Virtually all of the products caused our APC UPSs to reboot, which (fortunately) didn't affect anything but the management interfaces. So, be careful what IP addresses you scan and how the scans are run.

You'd think our networks wouldn't have gotten too far out of date in only two months, but these scanners had a lot to say. The winner by weight was eEye, which dumped a 180-page report on our desk, although McAfee won by count, telling us 537 different things about those four systems, 380 of which weren't specific vulnerabilities, but only informational items. Still, that left 84 critical vulnerabilities that McAfee wanted us to fix.

An obvious conclusion was that some vendors weren't doing a good job of data reduction. Yes, it's true that Adobe patch APSB10-14 covers 28 distinct vulnerabilities, but they're all fixed by a single patch, and treating that as 28 separate incidents -- as McAfee does -- simply encourages confusion.

Critical Watch had a similar issue, delivering all of the pieces of some of Microsoft's critical patches as separate elements, even though the remediation task was the same: install MS11-012 to fix five separately reported vulnerabilities. The pedantic security wonk might insist on knowing about each separate issue, but that's what drill-down reports are for. By default, this information should be combined into a more digestible format.

Some results were very cut-and-dried. For example, on our test of Windows systems, we had a list of security updates and patches that Microsoft Update gave us and we expected to see each of those patches in the list of vulnerabilities for each system. With McAfee, Qualys, and eEye, we found everything in our checklist in their scan results. Critical Watch, Lumension, and SAINT didn't catch everything.

If that checklist of known missing security patches was our entire test, it would have been easy to rank products. But each of the scanners had lots to tell us, and figuring out whether or not the information was relevant or worthwhile was tricky.

For example, Lumension told us that we were running an out-of-date version of SecureCRT (true) on our Windows systems and rated the vulnerability as "high" (probably over-kill). No other product we tested picked up on this. Does this mean that Lumension is better than the other scanners, which didn't catch this problem?

Well, yes, except that eEye found a circa-1999 protection setting on an obscure registry key that could be used by privileged users to further escalate their privileges during system boot. Does that make eEye better than the other scanners, which didn't identify the registry problem? You can go in this circle forever, as each product called out issues, mostly minor ones, that the others didn't.

Our Commenting Policies
Latest News
rssRss Feed
View more Latest News