Spam in the Wild, The Sequel

This time, we tested (almost) everyone

How big can a test get? We found out with our latest in-depth look at the anti-spam industry. Spam is still a huge problem, and there is an equally large market opportunity to fix it.

How big can a test get? We found out with our latest in-depth look at the anti-spam industry. Spam is still a huge problem, and there is an equally large market opportunity to fix it.

We invited every anti-spam vendor in our online Buyer's Guide to participate. While we expected to get eight to 10 vendors to sign up, 41 showed up. We tested them all for spam catch rate (including false-positive and false-negative rates), and performance and throughput (see charts).

Then we let the products speak for themselves. Out of the 36 that made it through the first round, we felt any product with a greater-than-90% spam catch rate and lower-than-1% false-positive rate should get a more in-depth evaluation. We still ended with a dozen excellent finalists, reflecting the growing maturity and commoditization of anti-spam products.

Analyzing the spam test results

Digging deeper into the accuracy and performance data.

Top spam fighters offer feature diversity

For the top of the heap of spam products, it's not what is good or bad that sets them apart. It's more a matter of what's different.

Adventures in spam testing

Testing routers and switches is easy. Frames go in, frames come out. With anti-spam products, nothing is ever easy.

Why our numbers work

You may notice our numbers are not as optimistic as the marketing literature from vendors' products. There are four reasons for this.

Where's SpamAssassin?

Although a few well-meaning souls volunteered to be the contacts for SpamAssassin, when it came time to test no one would step up to the plate and represent the product at a level that would make it competitive to the other enterprise-focused vendors.

Buyer's Guide: Anti-spam products

A complete buyer's guide of more than 130 anti-spam appliances, software and services.

How we did it

The methodology we used to test the various enterprise anti-spam products.

Pricing and featuresExcel spreadsheet)

Pricing and features data for the spam products we tested. (

Who got left out or opted out

Our criteria for inviting vendors into our test and who was not selected or turned us down.

What is a false positive?

With spam, suddenly everyone cares about statistics. For the first time, system administrators are buying software that openly admits that it doesn't work all the time.

Which comes first, anti-spam or anti-virus?

We used an anti-virus scanner to pre-scan messages for viruses and delete them before passing the message onto the anti-spam scanners we were testing. It raised the question - what is a best practice in the enterprise messaging space?

Insider Shootout: Best security tools for small business
Join the discussion
Be the first to comment on this article. Our Commenting Policies