Network vulnerability assessment management

Eight network scanning tools offer beefed-up management and remediation

Eight network scanning tools offer beefed-up management and remediation.

A vulnerability rated as a low risk this morning could turn into your worst nightmare tonight. To meet the ever-increasing speed with which exploits are written and propagated, traditional network-based vulnerability scanners have morphed into more full-scale vulnerability management products.

In our latest Clear Choice Test of eight products - assessing their accuracy in pinpointing holes in the network and their usefulness in addressing those vulnerabilities - we found vulnerability identification success rates are still low across the board and the scans can wreak havoc on wireless access points. They also can do damage to some printers, and can suck up network bandwidth and CPU utilization on target machines (see How we did it).

Getting a second scanning opinion

Citadel focuses on vulnerability remediation

How we did it

Archive of Network World reviews

Subscribe to the Product Review newsletter

Vulnerability remediation and tracking are the major management features added to these products since our last test, providing mechanisms to assign and alert administrators to new vulnerabilities. These additions range from providing vulnerability remediation information to offering full-blown ticketing systems that automatically verify if an issue has been fixed.

Business analysis features have been included in many products. With this functionality, assets can be given values - in terms of cash or business-critical value. How vulnerabilities potentially could affect business and give management a more accurate picture of the company's overall security posture can be correlated. A critical vulnerability on the core, Internet-facing system that generates revenue should be treated differently than a critical vulnerability on a system inside a test network that's isolated from the rest of the company, for example.

The companies that provided products and/or services for this test are Lockdown Networks, nCircle Network Security, PredatorWatch, Qualys, StillSecure, Tenable Network Security, TraceSecurity and Visionael. EEye Digital Security, Internet Security Systems, Foundstone, NetIQ, Bindview and Harris declined. We also tested Citadel's Hercules (see story) and Sunbelt Software (see story), but because they offer no scanning module or management features, respectively, we could not directly compare them.

Qualys' QualysGuard is our Clear Choice winner based on its accuracy and strong management capabilities. NCircle's IP360 comes in second, only slightly trailing Qualys in vulnerability identification and general ease of use. Visionael Enterprise Security Protector and Lockdown's Auditor also rose to the top based on their developing management capabilities.

QualysGuard 3.3

QualysGuardQualysGuard - one of the two vulnerability assessment services we tested - has a 1U appliance that sits on your network and lets Qualys scan your internal subnets. Setup is easy, and the quick start guide will have you scanning in no time. Because it is provided as a service, the Qualys team seamlessly adds the vulnerability checks.

Our discovery assessment focuses on how well the products find and identify systems, system software and services running on the network. Our accuracy measurement takes into account how well the product identified vulnerabilities that existed on a sample of lab systems (see "How we did it").

Qualys scored highest in our operating system identification checks and was the only product to correctly identify the wireless access point. It performed as well as any of the other products in the vulnerability accuracy tests, but still reported some false positives and false negatives. It did perform strongest among the products in identifying Windows system vulnerabilities, though.

Scan impact was low from a network perspective, but we did need to restart a Red Hat Enterprise system that became completely unresponsive after the scan.

Overall, QualysGuard is very flexible and easy to use. IT staff and/or corporate executives can be given varying levels of access to system groups and reports. Scan and report templates provide flexibility in the types of checks that are performed and how the results are viewed.

Remediation policies can be configured to automatically assign tickets in the Qualys ticketing system to defined individuals based on scan results. Qualys could improve on remediation if it added some preemptive notification mechanism to tell IT folks they have been assigned a remediation task.

In terms of providing some business analysis capabilities, Qualys lets you rank assets in terms of how critical they are to your business. A score is then provided in the summary based on your overall exposure level that can be weighted based on how critical the vulnerable asset might be.

One of the best features of QualysGuard is its mapping functionality, which provides a graphical representation of all the devices it discovers on your network. You can drill down on the map to identify the operating systems and services running on these devices, but can't see information on identified vulnerabilities from this vantage point. In addition to the mapping, we'd also like to see some sort of overview console that provides high-level information on the state of vulnerabilities on the network.

NCircle IP360 6.2

NCircle provided a central reporting server, VnE Manager, and scanning point, Device Profiler. With this tiered approach, nCircle runs in a more distributed model than some of the other products tested.

The IP360 provides the best business impact and risk-rating features, offering unparalleled levels of detail. Users can provide asset values for each host and calculate risk scores for each system based on the asset value. This value is a quantitative number, generally dollars, of the value of the asset to the company. As a consequence of this increased functionality, it is not as easy to use as some of the other products tested.

For system discovery, nCircle uses dynamic host discovery, its technique for continuously evaluating environments for new systems on the network. After running on the network for a few minutes, the system had found all the devices in the lab.

For operating system identification, nCircle joined Qualys as the only products to correctly identify the Cisco VPN Concentrator. But it missed a few key systems that most other products identified, including the FreeBSD 5.2 server and the Quantum Snap Server.

For vulnerability identification, nCircle consistently reported the smallest number of vulnerabilities, minimizing false positives, but potentially introducing some false negatives as well.

While nCircle's scan results might appear to include false negatives, following the remediation guidelines for identified vulnerabilities will address the known vulnerabilities in the system.

NCircle accrued the lowest network and system impact, with no identified issues or spikes in network traffic or CPU utilization.

One unique feature of the IP360 is its continuous scanning mode, which provides non-intrusive, back-to-back scans of the whole network or of only select segments.

This is ideal for critical systems or networks that need to be monitored at all times. NCircle provides a classic scanning model of scheduling scans, grouping systems and providing detailed user access.

NCircle takes a different approach in providing vulnerability remediation information. For the sample of vulnerabilities we reviewed, nCircle provided links to patched versions or specific patches for a variety of operating systems. In a few instances, the vulnerability remediation information did not match the specific vulnerability identified, although following the recommended course of action would in most cases have fixed the vulnerability because one patch would fix several issues.

Visionael Enterprise Security Protector

Visionael uses Nessus as its underlying scanning engine and focuses on providing some of the best vulnerability management functionality, such as a customizable portal for viewing security trending information.

Installing Visionael on Red Hat Enterprise worked well, although we'd like to see Visionael better secure the assessment server by default rather than leaving that up to the systems administrator.

Upon initial logon, Visionael provides the best portal functionality, allowing customization for each user and quick views of identified vulnerabilities, current risk level, trending and trouble ticket status.

There were a few issues in terms of system identification for the hosts on the lab network, namely system identification was not happening as we configured it. Working with support, we enabled the detailed operating system checks and reduced the concurrent threads from 200 to 20. With these changes in place, we got operating systems identification results, but they were not as detailed as we would like to see. For example, all Windows systems, regardless of version, reported back as "Windows."

For network and system impact, Visionael is quite loud. The scan locked up the wireless access point, bluescreened a Windows XP system and consumed 30% of the CPU on the monitored target system.

Viewing individual scan results provides an overview of identified vulnerabilities, with a breakout summary of the SANS Top 20, which is unique to this product. We would like to be able to drill down into the report directly from the vulnerability numbers reported in this overview screen.

The reporting module provides a wizard to create custom reports. But the customization options are so abundant that they are almost overwhelming.

The ticketing system is very strong, although tickets only can be auto-assigned for SANS20 or high-level vulnerabilities, which is fine if you prefer to do more detailed analysis on the other levels of vulnerabilities before tasking them out.

Visionael can auto-remediate identified vulnerabilities, but this functionality was not enabled in the license we received for testing.

For business analysis, Visionael provides strong trending information, executive reports and business rank, based on assigning systems one of four levels depending on how critical it is to your business.

Lockdown Auditor 3.0

Auditor provides the most intuitive management features, but lags a bit with its scanning engine.

Lockdown's 1U scanning appliance is the most intrusive, utilizing 40M bytes of network bandwidth and on average 40% CPU on the target system. Administrators do not have any options to change scan configuration settings.

It performed fairly well in terms of operating system identification, missing only two devices and not clearly distinguishing a few Windows versions. In terms of accuracy, Lockdown Auditor hit and/or missed to the same degree as most competitors.

Scan reporting is strong, providing a summary of identified vulnerabilities. We liked the job queue functionality, which shows the percent completion of each system being analyzed in the current scan.

In terms of management, Lockdown's user interface is the best of the products tested, combining graphics and a workflow very effectively. While it does not contain a specific portal, the initial logon window defaults to the report section. This provides an online version of the Executive Summary, which contains an overview of scan results and trending information.

For business impact analysis, the system provides a rating number based on scan results. You can assign critical values to specific systems, which then will be weighted more heavily when calculating the overall rating.

Lockdown's vulnerability notification capability was excellent and lets administrators define policies that trigger alerts. You can configure a policy that sends a page or SNMP trap and opens a ticket if a specific port was opened on a system.

Other unique lockdown features are the ability to encrypt e-mail that has account information using gpg and the ability to authenticate users against a corporate Lightweight Directory Access Protocol directory.

For remediation, it provides an excellent breakdown of problem, solution and resolution for identified vulnerabilities, including Common Vulnerabilities and Exposures numbers and links to related security advisories or remediation steps.

StillSecure VAM 4.0

StillSecure's VAM was a solid performer in both scanning and management. Setting up VAM on the vendor-supplied server was simple. The software automatically installed on the system from a CD when it booted up.

StillSecure's user interface is not intuitive and is difficult to navigate. The screen is often cluttered, making it difficult to identify specific information or tasks.

However, StillSecure performed fairly well in scan tests. In terms of operating system identification, it only missed a few of the network devices, such as the NetScreen-100 and the Cisco VPN Concentrator. It incorrectly identified the wireless access point as an ATM switch.

It performed well on scan impact analysis, providing no noticeable issues.

The scan report provides a generic list of vulnerability titles that you can drill down into for more details, although report navigation is a bit cumbersome.

VAM includes a robust ticketing system for tracking vulnerabilities, but it doesn't provide any business impact analysis functionality.

Reporting functionality in StillSecure is functional and offers some trending and executive report facilities. But it doesn't provide all the flexibility in other products.

Tenable Lightning 2.5, NeVo 2.0 and Nessus 2.0

The primary author of Nessus founded Tenable, so it's no surprise that Tenable's suite of products taps deeply into the Nessus base code to yield some unique features, such as Unix authentication for local vulnerability checks.

In addition to the Nessus active scanning engine, Tenable's Lightning product is the management console, and its NeVo product is the passive vulnerability scanner.

The Lightning/Nessus combination provides a very robust vulnerability search mechanism with the ability to search databases of identified vulnerabilities on almost any criteria. However, it doesn't include any mechanisms to control trouble ticketing or remediation functions or offer any business impact analysis.

In scanning tests, Tenable performed fairly well. In terms of operating system identification, it missed some of the network devices but performed the best of the Nessus-based products on scan accuracy. The overall scan impact was minimal.

The reporting module inside the console automatically generates a few reports, such as 30-, 60- and 90-day vulnerability details. You also can create custom reports from a selection of filter criteria.

One area that could be improved is in its vulnerability plug-in descriptions, which is what Nessus uses for its vulnerability checks. When trying to view the checks Nessus scans run, the drop-down box of signatures lists them by a nondescript ID, such as "CSCdp58462." These aren't very useful when trying to figure out what check is being performed.

Overall, Tenable has a very strong foundation, and we would like its vulnerability management functionality improved.

TraceSecurity's TraceAudit

TraceAudit is delivered as a service but also includes an ISO - a file that contains a complete image of a disk - and a hardened version of Red Hat that installs on a network system and provides TraceSecurity access to internal systems for scanning.

The results of your scans are sent over an encrypted channel to TraceSecurity's servers, with the results available from the Web-based management interface. The scan results are not encrypted when stored on TraceSecurity's servers, though.

Reports or general scan results do not include operating system information. Accuracy of scan results was in line with the rest of the products.

Scan impact was fairly low on the network, but it did cause a core dump on the HP print server.

The user interface is not intuitive from the start, but it becomes more usable once you understand the workflow. One oddity after the initial scan was that you couldn't view the results until you associate the systems to a group. If you launch a scan, the results should be automatically viewable through a default group or similar architecture.

TraceAudit doesn't have a formal ticketing system, but the company recommends customers use the system grouping functionality to manage vulnerabilities. While this might work for some organizations, it doesn't provide the full accountability and tracking we prefer in an enterprise vulnerability management product. It also doesn't include an overview portal/summary for reporting or the ability to filter out SANS 20 results. This product also doesn't offer any business impact analysis.

PredatorWatch Auditor 128 2.2

Predator Watch was a bit above average in its scanning features but really needs to bulk up its management capabilities to compete with this lot.

PredatorWatch runs on a small, square appliance that easily could fit on a desk. The software is available on a 1U appliance.

In our scanning tests, PredatorWatch was on par with several other products in that it missed some network devices and didn't differentiate between some versions of Windows. Scans locked up the wireless access point.

One function we couldn't get to work was launching an immediate scan. We would select a scan to start, but it never began. Scans would start fine when scheduled.

The management GUI is slow and unresponsive. We had requests processing for 30 seconds or more before the page hit the screen. The GUI is also difficult to navigate and not intuitive.

PredatorWatch doesn't provide a ticketing system, business impact analysis and user management functionality. It also only provides three reports - executive, management and administration - for each scan results set. Limited trending information is included from the previous scan, but we would like to see more custom report options and trending information.

PredatorWatch offers a unique feature with its compliance reports. Based on identified vulnerabilities, administrators can run reports to help identify weaknesses in Sarbanes-Oxley, Health Insurance Portability and Accountability Act and ISO 17799 compliance.


It's good news that vulnerability assessment tools are embracing vulnerability management functionality. Ticketing systems, business impact analysis, console dashboards and custom reporting options quickly are becoming standard features.

However, the number of false positive and false negative scan results still points out that vendors need to continue to refine their scanning engines. Users will benefit from strong management tools only if vendors make sure the vulnerabilities bubbling up to the management tools are complete, accurate and do not affect target system functionality.

QualysGuard 3.3 OVERALL RATING
Company: Qualys Cost: $67,500 for 1,000 servers, and $102,500 for 1,000 servers and 9,000 workstations. Pros: Excellent operating system identification; highly flexible scans and reports. Cons: No overview portal; no alerts sent on ticket assignment.
Enterprise Security Protector OVERALL RATING
Company: Visionael Cost: Starts at $15,000 for 1,000 devices and $120,000 for 10,000 devices. Pros: Detailed, customizable portal page; excellent vulnerability remediation information. Cons: Loud, intrusive scan; sensitive operating system identification.
Company: StillSecure Cost: $12,500 for 1,024 IP addresses and $45,000 for 10,048 nodes. Pro: Robust ticketing system. Con: GUI difficult to use.
Company: TraceSecurity Cost: $15,000 for 1,000 nodes; $39,000 for Class B network (approximately 65,000 nodes). Pro: Installs with hardened operating system. Cons: Poor user interface; scan data not encrypted on remote server.
Company: nCircle Network Security Cost: Starts at $60,000 for 1,000 nodes and $150,000 for 10,000 nodes. Pros: Non-intrusive scans; excellent impact analysis tools, continuous scanning. Con: Vulnerability descriptions need improvement.
Lockdown Auditor 3.0 OVERALL RATING
Company: Lockdown Networks Cost: $22,000 for 1,024 IP addresses and $178,000 for 10,048 IP addresses. Pros: Excellent GUI; intuitive workflow. Con: Scans very intrusive.
Lightning 2.5, Nessus 2.0.12, and NeVo 2.0 OVERALL RATING
Company: Tenable Network Security Cost: $27,600 for 1,000 nodes and $100,800 for 10,000 nodes; $12,000 for NeVo per installation. Pros: Passive scanning product; unique Nessus technology. Con: No trending information.
PredatorWatch Auditor 128 2.2 OVERALL RATING
Company: Predator Watch Cost: $16,000 for 1,000 nodes and $28,000 for 10,000 nodes. Pros: Offers regulatory compliance reports. Cons: GUI very slow to respond; minimal scan reporting options.
The breakdown Scan assessment (50% of overall score) Qualys nCircle Visionael Lockdown StillSecure Tenable Tracesecurity Predator Watch
Discovery 12.5% 5 4 3.5 3.5 4.5 4 3 4
Accuracy 12.5% 2.5 2.5 2.5 2.5 2.5 2.5 2.5 2.5
Scan ease of use12.5% 4.5 4.5 4.5 4 4 4 4 4
Scan reporting features 7.5% 5 5 4 4 4 4 3 2
Impact 5% 3.5 5 2.5 1.5 5 5 4 4
Management assessment (50% of overall score)
Remediation features 12.5% 4 4 5 5 3.5 1.5 2 2
Management reporting tools 12.5% 4.5 5 5 4.5 4 4.5 3 2
Business analysis features 12.5% 5 5 5 5 3 1 1 2
Documentation/ease of use 12.5% 5 4 4 5 3 4 3 1
TOTAL SCORE 4.36 4.25 4.11 4.06 3.61 3.36 2.74 2.54
Scoring Key: 5: Exceptional; 4: Very good; 3: Average; 2: Below average; 1: Consistently subpar

Learn more about this topic

Andress is president of ArcSec Technologies, a security company focusing on product reviews and analysis. She can be reached at

NW Lab Alliance

Andress is also a member of the Network World Lab Alliance, a cooperative of the premier reviewers in the network industry, each bringing to bear years of practical experience on every review. For more Lab Alliance information, including what it takes to become a member, go to

From CSO: 7 security mistakes people make with their mobile device
You Might Like
Join the discussion
Be the first to comment on this article. Our Commenting Policies