Vulnerability alerting services

When a new security vulnerability turns up, you need to know when it was released, which products it affects, what it possibly could do to your network and how you can preemptively address it.

Now multiply that by 889 – the number of vulnerability alerts issues in 2004 – and you’ve got a serious pile of data that needs to be sorted and analyzed by knowledgeable security staff that you may, or may not, have at your fingertips.

Example Symantec vulnerability report (PDF)

Audio: Behind the scenes at PivX

How we did it

Archive of Network World tests

Subscribe to the Network Product Test Results newsletter

In our test of vulnerability alerting services from Co-Logic Security, Cybertrust, PivX Solutions, SecurityMob, SecurityGlobal.net, Secunia and Symantec, we assessed how well each can provide timely, relevant, accurate and useful information directly to your in-box.

Overall we found these services to be very useful because they help filter the myriad alerts. With these services in place, IT security personnel can then focus on remediation plans rather than combing through mailing lists and vendor sites culling for newly issued alerts.

When the data analysis dust settled in our own tests, Symantec’s DeepSight Alert Service – which the company picked up in its acquisition of SecurityFocus last year – came out as our Clear Choice winner. It provides the most delivery options, very detailed reports, detailed alert category configuration and quick response times.

Cybertrust’s Alert Manager – a product coming from the merger of TruSecure and Betrusted, completed in November – was a close second, falling behind on alert delivery options, but standing very strong on alert information and format.

We’d also put the services from Secunia, PivX and SecurityGlobal on a short list. Secunia’s Security Manager is one of the least expensive services in this test and is very consistent in its alert delivery. The interface driving PivX’s ThreatFocus Diligence is intuitive and therefore makes the service very easy to use. And SecurityGlobal’s SecurityTracker is very effectively focused in its one mission: getting alerts pushed out to its customers.

The services generally differentiate themselves in terms of how each collects and interprets vulnerability information. Some services, such as Co-Logic Security’s E-Secure-IT and SecurityMob, act solely as data aggregators. They compile information gathered from hundreds of sources – mailing lists, vendor announcements, Web site scouring and direct messages from vulnerability researchers – and then send pertinent alerts your way. E-Secure-IT also provides a collection of security-focused articles from hundreds of different sources. As a security professional, it is a big bonus to have all this information collated in one place.

The other services tested focus on providing information and analysis for vulnerabilities to help users understand how quickly they need to react to a new announcement. Symantec and Cybertrust focus on broad coverage and perform their own analysis and write-ups. And both perform their own testing on reported vulnerabilities, provide detailed threat analysis and remediation options, and provide links to associated vendor patches. Security Manager, SecurityTracker and ThreatFocus Diligence offer their own descriptions but generally are not as in-depth as Symantec and Cybertrust.

These alerting services also vary in how high they raise the red flag on any particular alert. There seems to be a general tendency to set the risk and severity ratings to accommodate worst-case scenarios, which sometimes can be misleading. We advise, based on that observation, that users be wary about basing patch deployment decisions solely on these services. There is always some internal analysis any organization must perform to see how fast it needs to react to a given alert. For example, an alert announcing an Apache server vulnerability might not be critical to an organization that only runs Apache on its intranet Web server. But a company running Apache on all its Internet-facing Web servers might need to react immediately.

SecurityTracker doesn’t provide any risk information, leaving the rating decisions solely in the hands of the organization. But the company says it is adding functionality to the service in the near future that will help organizations set their own internal risk ratings. DeepSight Alert Service stands at the opposite end of this spectrum, providing analysis and a 1-to-10 rating in the areas of severity, impact, ease of exploit and credibility. The remaining products fall somewhere in the middle of these two extremes.

For this Clear Choice Test, we focused on four main performance areas: alert information, or the details provided within an alert distributed by the service; alert delivery, which includes timing, methods and format; alert management, focusing on the GUI used to access and configure the service, general ease of use, and documentation; and alert coverage, which looks at how the service identifies new security issues and how many products it covers in its research.

What can you tell me?

In the alert information category, we reviewed the classification of vulnerabilities, the description provided for the issue and the suggested remediation plan.

Vendors were split between providing an all-in-one rating (where they would simply give the alert a high, medium or low tag) and breaking out different areas such as severity or possible impact. The latter offers more detail, which can be very useful when performing further analysis.

DeepSight Alert Service provides the best overall information, breaking risk ratings into severity, impact, ease of exploit, and credibility of alert, all on a scale of 1 to 10. (See full Symantec report) Alert Manager breaks things out into urgency, credibility and severity based on a scale of 1 to 5. The remaining services used the all-in-one approach, with Security Manager providing the best breakdown on a five-level scale. SecurityMob uses a four-level scale, ThreatFocus Diligence uses three levels, and E-Secure-IT issues red and yellow alerts for significant threats.

Our problem description assessment focuses on the value of the information included within the advisory to explain the issue.

DeepSight Alert Service and Alert Manager stand out in this area, providing focused reports that include a basic description, detailed impact description, technical description, attack scenarios and exploit links (if available). Alert Manager adds a comments section where their analysts may provide comments on the security issue at hand. In our test, these notes provided little known details on the vulnerability and high-level discussions on how to prevent vulnerabilities from being exploited.

SecurityTracker, Security Manager and ThreatFocus Diligence also write descriptions, but we found them in general not to be consistent in the level of detail they offer.

One feature we would like to see added across all services is mapping alerts to Common Vulnerabilities and Exposures (CVE) entries on a consistent basis. This mapping would help tie reports from internal vulnerability scans and intrusion-detection system reports to the alerts posted by these services.

In our assessment, the most important component of the alerts is the recommended remediation actions, advising organizations how they should respond to the issue at hand.

DeepSight Alert Service provides a comprehensive list of links to vendor patches and workarounds. While useful, combing through this long list can be cumbersome. Alert Manager discusses “safeguards” for preventing this vulnerability in the first place, and provides links to vendor announcements and patches. We found that with both of these services, many of the recommended safeguards or workarounds reflect general security principals that companies should have in place as part of their basic infrastructure.

SecurityTracker does not perform its own remediation analysis, so users of this service will have to rely on the remediation actions included by the entity that originally discovered and reported the vulnerability or rely on a vendor announcement.

Security Manager goes back and forth between providing comprehensive links to vendor patches and just pointing to the original source. For example, in the mod_include Apache vulnerability we reviewed in detail, Security Manager only referred to the Apache Web site to get the latest source and not any of the other vendors that had issued updated Apache packages.

ThreatFocus Diligence also provides links to patches and periodically includes workaround recommendations. These recommendations are specific to the vulnerability and not references to general security practices.

E-Secure-IT and SecurityMob just pass on the original alert.

How can you reach me?

For alert delivery, we looked at the method and format options used by the service and the timeliness of the announcement. For delivery methods, DeepSight Alert Service stands out with the most options, including e-mail, Short Message Service (SMS) mobile text messaging, fax and voice. All services provided e-mail alerts and about half offered SMS alerts. SecurityTracker is the only other service we saw that provided voice alerts, which occurs through a third-party partner.

Most alerts are delivered in plain text, but some offer XML and/or HTML. Alert Manager sends the best formatted alerts, with HTML alerts issued in a column newsletter format. The text is sometimes hard to follow if the discussion moves to the next page, because you expect it to continue in the next column, not the next page.

DeepSight Alert Service is the only service currently offering PDF delivery. The reports are clear and easy to read. Security Manager delivers alerts in text format, but includes more information and has a better presentation for the same alerts on its Web site. For example, alerts that have CVE entries associated with them are only shown on the Web site. ThreatFocus Diligence offers text alerts, including how the alert is displayed in the Web console. We would like to see a print option for these ThreatFocus reports.

How quickly can you tell me?

Timeliness of alert delivery is critical. For our sample selection of alerts, we made sure to include vendor announcements, issues posted to mailing lists, older alerts with updates and open source software. DeepSight Alert Service was always the fastest, sometimes issuing alerts days before the other services. SecurityMob alerts also were always issued at the front of the pack. Across the board, alerts issued from vendor bulletins, such as the Sun and Cisco advisories used for our testing, generally were issued the day the vendor posted bulletins.

The most interesting timeline was for the IFRAME vulnerability posted regarding Internet Explorer. DeepSight Alert Service issued a bulletin on Oct. 25, 2004, the day the original issue was reported on the Bugtraq and Full-Disclosure mailing lists.

The remaining services issued alerts in early November when the proof of concept exploit code was posted to several security mailing lists.

In another instance, SecurityTracker did not issue an alert for the Sun Samba bulletin contained in our test sample, as it provided no new information over the original July 2004 announcement of the issue. This happened in spite of Sun issuing a security bulletin discussing the vulnerability on Oct. 26, 2004. All other services alerted on the bulletin within a day of its release.

Security Manager delivered its alerts to us in batches during the night. We consistently received these alerts the night the vulnerability was announced or early the next morning. Alert Manager generally issued the alert the day the vulnerability was announced, but was a few hours later than the other services that issued that day. E-Secure-IT was fast on major vendor bulletins, but very slow on issues reported on mailing lists or in open source tools.

Managing what I see

Our alert management tests centered heavily on the GUI used to configure alert profiles and view alerts, and takes into account general ease of use and documentation.

DeepSight Alert Service employs a three-step method for alert configuration. You set up the delivery method, establish the desired technology lists and establish your monitors. The monitors are the vehicles that combine the delivery methods and product lists into an alert profile. Product selection for alerts is provided by an excellent interface that starts you off by choosing a product or vendor and narrowing the scope from there, such as limiting to only a specific version of a product. Inclusions and exclusions can be identified during any step, making for a very flexible interface.

Alert Manager has a similar setup for product selection, but the interface is not as smooth. To set up alerts, you create a product set and add products to these profiles. E-Secure-IT provides a plethora of information on advisories and security-related articles, more than any other service, but the interface is not very elegant. The screen is very cluttered, and everything is based on dynamically generated lists. If you have a lot of areas you are watching, this list can be very long.

SecurityTracker focuses on simplicity. The management interface is clean and simple, but product selection is one very long page of check boxes. It’s difficult to find anything, so we usually resorted to using the search mechanism in our browser to find what we were looking for. Look-up functionality and the ability to select specific versions of products would be a nice addition.

Security Manager takes a slightly different approach, requiring you set up information for actual servers, not just products. You select the operating system and any applications running on the system, and set the alert method. Secunia also offers the option to run vulnerability assessment scans (using Nessus) on externally facing IP addresses.

SecurityMob has a very clean, elegant management interface. It provides a mechanism called the “quick build profile” that presents a list of pre-built profiles for alerts, such as Windows and virus alerts. Adding more pre-built profiles, especially a Unix option, would help differentiate the SecurityMob service. To manually select products to add to your alert profile, you must expand through a tree-based list, grouped by product function. A search mechanism is not included to find a specific product.

ThreatFocus Diligence also needs to add a search mechanism to make it easy to find products to alert on. You currently scroll through a list of vendors and then select the available product category to monitor. Selecting specific versions also would be useful. ThreatFocus Diligence also includes assessment scans.

For general ease of use, SecurityMob, SecurityTracker and ThreatFocus Diligence were the most intuitive. DeepSight Alert Service was easy to use, but its complexity requires a little guidance from the documentation at the beginning. Our experience with Alert Manager was very similar and required us to read through the documentation to understand how to set up profiles and alerts. E-Secure-IT provides so much information that looking at the interface is a bit daunting at the beginning. While you get used to it, the overall experience of this service could be greatly improved with a new Web interface.

While documentation is not as critical with vulnerability alert services as it is with other security products, you still need a mechanism to answer your questions as you use the service. DeepSight Alert Service and Alert Manager provide the best documentation with user guides and online help. E-Secure-IT also provided hard copy documentation to help understand how to best use the service. ThreatFocus Diligence includes a strong online help section, but SecurityTracker, Security Manager and SecurityMob provide minimal online help for the management console.

What products can you tell me about?

An alert service is not useful if it does not provide valuable information on the products you are interested in. To evaluate how well each service hits this mark, we generated a list of products we wanted to monitor and attempted to set up the proper alert profile for our environment.

SecurityMob and ThreatFocus were the only two that did not cover all the products in our test bed. Both did not cover BEA Systems’ WebLogic or F5 Networks products. SecurityMob did not support monitors for the Brocade storage switch, the NetScreen Technologies Firewall or the Cisco VPN Concentrator 3000 in our test bed. ThreatFocus did not monitor the SnapServer, Network Appliance storage device or Sophos anti-virus software in our test.

From these tests, we can glean that these services can help make corporate security teams more efficient, by letting them outsource research services to companies that define this work as their own core competency.

The main surprise in our testing was the differences in the timing of alert delivery across the services. We expected them to be closer together and not have some services days, or even weeks, behind issuing alerts.

In the end, our testing showed Symantec’s DeepSight Alert Services to be the best all-around service, with Cybertrust’s Alert Manager following very close behind, but keep in mind that you also pay a premium for the added features. SecurityTracker, ThreatFocus Diligence and Secunia’s Security Manager are very solid services that can come in at a lower price for the cost-conscious organization. E-Secure-IT and SecurityMob function mostly as content aggregators, passing on original source material without any additional analysis.

Regardless of the price, the key when selecting a service is to ensure it is compatible with your environment – how you want to receive alerts, information you want to receive in your alerts and, most importantly, that it covers all of your products.