Network scanners pinpoint problems
EEye's Retina wins our Blue Ribbon Award for speed and quick fix features.
In the past, there hasn't been much good news about the state of vulnerability-assessment scanners. Their reputation has been plagued with false positive reports, lack of scalability, lagging updates and inadequate reporting tools.
While some areas still need a bit of improvement, vulnerability scanners have useful tools for helping network professionals identify potential vulnerabilities and security. However, we also found that many of these products may have trouble scaling to fit the requirements of enterprise networks.
In our testing, we reviewed products from eEye Digital Security, Nessus, Symantec, Internet Security Systems, NetIQ, Network Associates, PatchLink and Harris. Cisco and BindView declined to participate.
We evaluated how each identified our network vulnerabilities; what resources it required to run and then scale to a larger network; its reporting tools; what it offered as security recommendations and autofix features; and installation and ease of use.
How we did it
Holes in your network
Vulnerability-assessment services on the rise
Interactive Buyer's Guide chart
Scorecard and Net Results (chart)
Testing scanner performance (chart)
Vulnerabilities tested for the Network World network-based scanner review (chart)
EEye Digital Security's Retina is the Blue Ribbon Award winner. Harris' Security Threat Avoidance Technology (STAT) Scanner was a close second, but it fell a bit short in the ease-of-use category.
PatchLink's Update product was impressive, but its support for Windows-only systems lowered its score. In the near future, PatchLink will support Red Hat Linux, Solaris, AIX and NetWare 5.x, the company says. We were impressed that the vulnerability-assessment aspect of PatchLink Update occurs behind the scenes. The only information you see is the results of the analysis, which informs you of all the patches that need to be installed on each system.
ISS's Internet Scanner and the open source scanner Nessus made good showings. Both provided good analysis overall, but performed slower than STAT Scanner and Retina. And both missed a few more vulnerabilities than the top performers in our test.
Symantec's NetRecon, NetIQ's Security Analyzer and Network Associates' Distributed CyberCop Scanner had several issues that held them back in our tests. NetRecon performed very slowly and identified vulnerabilities for services and configurations that were not running on the network. While NetIQ identified quite a few vulnerabilities on our systems, it missed the big ones, including one of the major known Internet Information Server 5.0 vulnerabilities. Its performance also lagged when scanning more than a handful of systems.
Distributed CyberCop Scanner takes a good approach to scanning enterprise networks by distributing the load to multiple servers, but its reliance on the ePolicy Orchestrator management console is frustrating.
The most important component of our testing checked to see how accurately the scanners identify vulnerabilities. Using our controlled test network (see How we did it), we selected 15 vulnerabilities we knew existed on the test systems (see the chart to find what vulnerabilities we looked for specifically and whether the product identified them). We selected a combination of Linux and Windows vulnerabilities and a few system configuration options, such as anonymous FTP and allowing null sessions, that do not follow generally defined security best practices. The vulnerabilities we selected covered a range of services, including Windows domain issues, FTP servers, Sendmail servers, Web servers and Secure Shell (SSH) devices.
The results were a bit surprising. All products recommended we disable the chargen/echo service, anonymous FTP, and ability to create null sessions on our systems, which is good. Enabling either of these services and enabling null sessions on Windows systems goes against security best practices.
Retina and STAT Scanner found the most of the 15 vulnerabilities on our target list with Retina missing only two Linux holes (SSH and wu-ftpd) and STAT Scanner missing one Windows (MS01-005) and a few Linux (sendmail, wu-ftpd and SSH) holes. Both products also provided information on important registry key and account policy setting changes. PatchLink Update also identified all the patches that needed to be installed on the Windows systems.
Overall, STAT Scanner provided the most comprehensive and accurate list of the vulnerabilities on our network. STAT Scanner requires a SSH connection to Unix systems and it requires that your administrator have a user ID and password to scan settings. The documentation and graphical user interface (GUI) say this account must have root access, but that is not the case. An ordinary user account works. Future documentation and the next GUI release will correct this issue.
Nessus and Internet Scanner performed well, identifying most of the Linux issues and the major Windows vulnerabilities. NetRecon only identified a handful of our known vulnerabilities and a few that were not even present on the system. For one Linux system, NetRecon told us we had some Apache vulnerabilities, and Apache was not even running on that system. NetRecon also told us our Cisco router was infected with the Girlfriend and Subseven Trojans when they were not present.
Security Analyzer performed the worst, identifying only our configuration issues but failing to accurately report any service-related vulnerabilities. All the 2,065 vulnerabilities identified by the product were host configurations such as file and directory permissions, registry permissions and account policies. Security Analyzer will also not identify Unix/Linux vulnerabilities without an agent installed on the system. Even when we installed this agent, Security Analyzer did not identify any of the vulnerabilities on our list.
Fixing the problems
After identifying the vulnerabilities on each system, these scanners generally provided some tips on how to correct the problems. The autofix feature is where PatchLink Update thrives, providing a comprehensive assessment and patch management feature that lets administrators automate and centralize Windows patch distribution and installation.
While not as comprehensive as the PatchLink offering, Retina and STAT Scanner provided a means of automatically and remotely fixing some registry and permissions problems. In a future release, Retina plans to offer a means of installing Windows patches.
Internet Scanner does not provide any autofix functionality, but it provides the best step-by-step instructions to fix identified vulnerabilities for Unix and Windows platforms.
For vulnerability descriptions and identification, each product provides different sets of data. Retina provides the most information, including Bugtraq IDs, Common Vulnerabilities and Exposures (CVE) numbers, and vendor patch numbers where available. ISS only lists the CVE number. NetRecon uses the CERT Advisory number. STAT Scanner uses the Q ID number for Microsoft bulletins, and PatchLink Update uses the Microsoft Bulletin numbers. Comparing the output of all these scanners to identify exactly what vulnerabilities they are describing is a cumbersome and arduous process.
For identifying patch locations, Retina and STAT Scanner are again at the head of the class. They provide direct links to the patch location or a link to the program's Web site, such as www.sendmail.org for Sendmail vulnerabilities. STAT Scanner takes this one step further for Red Hat Linux systems by providing direct links to Red Hat's packaged management platform.
After vulnerability identification, reporting is the next most important feature of any vulnerability scanner. It's one thing to find the holes on the network, it's another to present that information to the user in a timely and organized format. Reporting is fairly standard across the assessment scanners we tested, but a few issues stood out.
All scanners provided at least HTML report output. NetRecon and Internet Scanner provided PDF exports, which is very nice. We would have liked this feature in all the products.
But STAT Scanner offers the widest range of reporting options by providing a long list of potential reports. If there is a specific way you would like the information displayed, STAT Scanner will have the report format available. Internet Scanner and NetRecon also provided a decent range of report options, including executive summaries and detailed technical reports.
PatchLink Update provided online reports detailing what patches are installed on a system and which systems need specific patches. These reports are only available online, though. We would like to have the option to export and/or print reports.
Security Analyzer's report can be exported to HTML or Word. However, it is cumbersome to review. The information is jumbled and not presented in an easy-to-read format.
Installation and ease of use
All the products, with the exception of Nessus, used standard Windows installer programs. After installation, STAT Scanner, Retina, Internet Scanner, NetRecon and Security Analyzer were ready to go. PatchLink Update required agent installations on each machine, and CyberCop Scanner needed to be configured through ePolicy Orchestrator and have agents deployed to systems that will perform the scanning.
To initiate a scan, most products only needed an IP address, range of IP addresses and domain name to start. System identification is generally built in to the scanner, with the exception of STAT Scanner. This uses a third-party application, What's Up Gold by Ipswitch, to identify systems on the network.
The Nessus installation process went smoothly. We followed the instructions found at www.nessus.org, installing the necessary libraries and then running the Nessus installation shell script.
Retina and Nessus use Nmap, the open source fingerprinting tool, to identify systems running on the network, what ports are open and what operating system is running. Other scanners use proprietary techniques for identification.
NetRecon, Nessus and Retina did a good job identifying devices on the network, including the NetScreen firewall appliance and its corresponding management server, SNAP Network Attached Storage device, and a Hewlett-Packard printer. STAT Scanner, ISS and Security Analyzer only identified the Windows and Linux systems on the network. They found the active IP addresses for the other devices, but couldn't identify them.
Updating each scanner to identify the latest vulnerabilities was simple across the board. Out of the box, all the products, except Nessus, provided the ability to automatically download updates over the Internet. Security Analyzer actually checks each time the product is started and alerts you that updates are available. Symantec's process is the smoothest, using its Live Update infrastructure.
Nessus relies on the Linux community to provide vulnerability updates. Several individuals have written scripts that will go out to the Internet to download and install the latest vulnerability signatures. Once this process is set up, updates can be seamless and automatic.
Update availability varies from vendor to vendor.
They all say they release updates for major vulnerabilities as soon as they are available, which can be a few hours to several days or weeks. STAT Scanner releases new updates on the eighth day of each month. NetRecon releases updates every other week.
Retina has the best user interface with its slick colors and intuitive layout. STAT Scanner provided a lot of functionality in its product, and its interface shows that because it is a bit cluttered with information and small icons.
The Security Analyzer interface is the same as the one used by WebTrends reporting software, so many users may find this product strangely familiar. Distributed CyberCop Scanner uses the ePolicy Orchestrator for management, and was difficult to navigate and use.
The Nessus interface runs in X Windows and is a simple, low-frills application. The PatchLink Update interface is a slick Web-based interface that anyone familiar with any Web-based administration tool will find easy to use.
Retina is lightning fast, scanning our 12-system test network in less than 5 minutes. STAT Scanner also performed well, taking about 20 minutes. Internet Scanner took the longest in our first round of tests, having to be stopped after eight hours because it hung on the analysis of the management IP for the NetScreen firewall. After removing NetScreen from the test, Internet Scanner took two hours to complete.
After testing in our self-contained 12-system lab network, we ventured out into the real world to see how these products would act on more enterprise-sized networks. We started slow, scanning a network with 47 active Windows and Linux systems. Retina still flew through the process, completing the Class C scan in about 15 minutes. Security Analyzer never finished.
After scanning about 20 systems, Security Analyzer overloaded the system resources and could not finish. NetIQ tech support told us we needed more RAM. In our experience, the system necessary to scan a network with the NetIQ product would be much bigger than any organization would be willing to purchase.
We then moved on to a larger private network that contained approximately 500 systems. Retina still proved to be the fastest, completing the scan in a little more than two hours.
Vulnerability-assessment scanners are improving, but many are still spotty with vulnerability identification, and they have not been developed to efficiently scan large networks.
Retina and STAT Scanner stood above the rest of the crowd, but they missed a few vulnerabilities on our list. As new products are released, it will be interested to see whether vulnerability identification and scalability improve to better suit the enterprise.
Andress is a network security engineer at TiVo and a frequent contributor to many publications. She has also authored several books, including Surviving Security. Andress is also active on the conference circuit, speaking at Black Hat, NetWorld+ Interop, and numerous other conferences. She can be reached at firstname.lastname@example.org
Holes in your network
Vulnerability-assessment tools edge toward usefulness in large networks.
Vulnerability-assessment services on the rise
Like most markets these days, the vulnerability-assessment market has a new services-based component.
How we did it
Our testing methods explained.
Testing scanner performance (chart)
Interactive Buyer's Guide chart
Search for the vulnerability-assessment scanner that fits your network best.