Test: Spam in the wild - Network World

Skip Links

DNSstuff.com
Get information about your IP
IP Information
50+ On-demand DNS and network tools

Software

Videos

rssRss Feed
Get instant email notification when white papers, webcasts, executive guides are added to our library.  Stay informed and up-to-date with the latest on IT Technologies with Network World's Resource Alerts.

Additional Resources

RSS

FEATURED WHITEPAPERS

Edison Group TCO White Paper HP

Edison analysts put the management software of an HP EVA system through a series of typical day-to-day storage management tasks. The same tasks were also evaluated on similar systems from NetApp and EMC. This study demonstrates how the superior user interface and virtualization offered by the HP EVA storage system can provide organizations with the benefits of higher administrative efficiency combined with the potential ability to utilize less expensive human resources.

RSS

FEATURED REPORTS

Executive Guide: Storage Heats Up HP

Get the latest on storage technologies that allow IT professionals to better cope with new IT demands. Learn how storage technologies can help you successfully tackle e-Discover, regulatory compliance, green data center initiatives and the data explosion. Get all the details now.

Test: Spam in the wild

We throw real traffic at 16 anti-spam products.
By Joel Snyder , Network World , 09/15/2003
  • Social Web 
  • Email 
  • Feedback 
  • Close

Practically every vendor on the planet claims to be able to solve your spam woes. So we tested 16 products on a live production network to see who could back those claims. For the entire month of June, we threw a live mail stream, spam and all, at the products to see who could survive the spam onslaught, and who would choke.

Estimates of the amount of unwanted e-mail range from 40% to 75%, but we can give you an exact percentage - 69%. That's how much spam we saw during the month of June. And things are getting worse, not better - in a similar Network World test we ran in February (see review), only 50% of the mail stream was spam.

Fortunately, good products prevailed, and can help you significantly reduce your spam problem. With a very broad field, including service-based mail filters, appliances and traditional software on Unix and Windows, network managers should be able to solve their spam problem with a minimum of disruption - to the accolades of their users.

How well do they work?

We tested mail-filtering gateways by feeding them an e-mail stream in real time, as it came into our labs (see "How we did it"). Each product received two scores. The first score, sensitivity, measures how well the filter identified spam. A perfect score would be 100%. The second score is the false-positive rate, the ability of the filter to make sure that non-spam messages do not get tagged as spam. A perfect false-positive rate would be 0%. (For more about the different ways to measure a spam filter, see "Spam and statistics".)

Spam filtering is such that a high sensitivity naturally also would have a high false-positive rate. Similarly, a low false-positive rate might let a lot of spam through. We feel that enterprise network managers would be more concerned with false positives, so we asked the vendors to tune their products for a false-positive rate of about 1%.

Products from seven companies - CloudmarkCorvigoMailFrontierMX LogicPostiniTrend Micro and Tumbleweed Communications  - met our 1% requirement.

To identify the top products in filtering spam, we looked for a sensitivity rate of at least 80%. Products from seven companies also met that level - ActiveState, Cloudmark, Computer Mail Services, MailFrontier, Postini, Singlefin  and Tumbleweed. (For complete results, see graphic.)

Combining these lists gives us the top overall performers: Cloudmark's Authority, MailFrontier's Anti-Spam Gateway (ASG), Postini's Perimeter Manager and Tumbleweed's Messaging Management System (MMS).

Click to see: Accuracy and False Positives

Accuracy and False Positives

Of course, your results will vary, depending on your own message-stream characteristics and how well you tune the products. For example, Postini's spam-detection engine is at the heart of Trend Micro's recently released Spam Prevention Service (SPS). However, we got very different results with the two products, largely because Postini officials told us to tune their product using one set of numbers, while the Trend Micro team gave us a different set. This resulted in both a higher false-positive rate and lower spam sensitivity for Trend Micro.

Many vendors predicted that their false-positive rate would be much lower than 1%. Corvigo's CTO said its customers report a false-positive rate between 10 and 100 times better than our tests showed. That's easy to understand, because most of the false positives we saw fell into the category of "mail that wouldn't be missed by users," such as news stories forwarded by friends, e-mail from online merchants and postings to mailing lists. For example, Postini, which had the lowest false-positive rate, missed 28 messages it marked as spam. Of those, only five were messages we wanted to see. If we hadn't been combing our mail carefully, we wouldn't have noticed those messages as missing.

Some false positives were understandable, but regrettable. A message with the subject line "IOS fw guru" looked like spam to many of the filters, but turned out to be a job offer for our test lab.

Tuning to improve performance

Most of the products can be tuned to increase sensitivity and decrease false positives, but how this tuning is accomplished and who is responsible for it makes all the difference. There are two main tools used in tuning mail filters. First is the threshold that determines whether a message is spam. The best products offered a series of levels, often expressed as percentages, based on its guess as to whether a message is spam.

1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |  Next >
Comments (1)
Login
Forgot your account info?

RE: Test: Spam in the wildBy meatpieandtatters on August 27, 2007, 1:33 pmDid you test MIPSpace?

Reply | Read entire comment

View all comments

Add comment
Anonymous comments subject to moderator approval. Register here for member benefits.
Have a NetworkWorld account? Log in here. Register now for a free account.
First Name
Last Name
E-mail
Zip Code
IT Buyer's Guides

View All Buyer's Guides