Spam in the Wild, The Sequel
This time, we tested (almost) everyone
How big can a test get? We found out with our latest in-depth look at the anti-spam industry. Spam is still a huge problem, and there is an equally large market opportunity to fix it.
How big can a test get? We found out with our latest in-depth look at the anti-spam industry. Spam is still a huge problem, and there is an equally large market opportunity to fix it.
We invited every anti-spam vendor in our online Buyer's Guide to participate. While we expected to get eight to 10 vendors to sign up, 41 showed up. We tested them all for spam catch rate (including false-positive and false-negative rates), and performance and throughput (see charts).
Then we let the products speak for themselves. Out of the 36 that made it through the first round, we felt any product with a greater-than-90% spam catch rate and lower-than-1% false-positive rate should get a more in-depth evaluation. We still ended with a dozen excellent finalists, reflecting the growing maturity and commoditization of anti-spam products.
Analyzing the spam test results
Digging deeper into the accuracy and performance data.
Top spam fighters offer feature diversity
For the top of the heap of spam products, it's not what is good or bad that sets them apart. It's more a matter of what's different.
Testing routers and switches is easy. Frames go in, frames come out. With anti-spam products, nothing is ever easy.
You may notice our numbers are not as optimistic as the marketing literature from vendors' products. There are four reasons for this.
Although a few well-meaning souls volunteered to be the contacts for SpamAssassin, when it came time to test no one would step up to the plate and represent the product at a level that would make it competitive to the other enterprise-focused vendors.
Buyer's Guide: Anti-spam products
A complete buyer's guide of more than 130 anti-spam appliances, software and services.
The methodology we used to test the various enterprise anti-spam products.
Pricing and featuresExcel spreadsheet)
Pricing and features data for the spam products we tested. (
Our criteria for inviting vendors into our test and who was not selected or turned us down.
With spam, suddenly everyone cares about statistics. For the first time, system administrators are buying software that openly admits that it doesn't work all the time.
Which comes first, anti-spam or anti-virus?
We used an anti-virus scanner to pre-scan messages for viruses and delete them before passing the message onto the anti-spam scanners we were testing. It raised the question - what is a best practice in the enterprise messaging space?
Copyright © 2004 IDG Communications, Inc.