How we did it

We installed the enterprise anti-spam products by putting them in a production e-mail environment for about one month. To accommodate the large number of participants, we used two high-end, dual-Xeon servers loaned to us by HP, along with EMC's VMware ESX server. We installed 19 products within 19 separate VMware images on the HP servers, using whatever operating system the vendor recommended. For the nine appliances, vendors shipped us their hardware, which we installed in our machine room. To handle these systems, we used an AMX5100 KVM system from Avocent. Finally, 10 products were service-based, so we forwarded our mail through a dual-DS3 Internet connection to those providers.

Because of the scope of the test, we installed the products and attempted to debug any installation problems with the vendors over a two-month period. Etherpeek, provided to us by WildPackets, was invaluable in debugging problems with several vendors' SMTP implementations. We also were indebted to Microsoft for providing copies of Windows 2000 and Windows 2003 needed to test several products. During this time, we sent a subset of the Opus One incoming mail stream to each product to aid in testing and tuning. All vendors who requested it had full access to their test systems for tuning, configuration and debugging purposes.

One week before testing started, we reconfigured Opus One's corporate mail servers so that mail would come directly from the Internet to Opus One's mail systems. Because this was an anti-spam test, we ran the message through a virus scanner and discarded messages with viruses in them.

Pre-scanning the e-mail for viruses was important. Because nearly 100% of the virus-infected e-mail is actually created by mass-mailing worms (rather than true messages from people with infected attachments), many anti-spam products are beginning to treat mass-mail worms as spam. We didn't want the complication of dealing with differing interpretations of certain kinds of viruses as spam, so we pulled them out of the mail stream where possible.

We also found that some anti-spam products consider the double-bounces that these viruses can cause as spam, even through they are not virus-infected, and not mass-mailed. To head off disagreements on whether those messages are spam, we dropped approximately 100 obvious double-bounces from our test data and didn't count them in the final statistics. But we did discover that some products are too aggressive in deleting all non-delivery report (NDR) mail, including notifications from our own mail servers about real messages that that had failed delivery. Those clearly erroneous false positives were counted against products that tagged them, incorrectly, as spam.

Once we had cleaned the incoming mail stream, we turned it around and simultaneously re-fed the stream to each of the participants in very close to real time. An important part of our test was that each of the products in the final test was seeing the mail as close to the time we received it (when possible). Later testing with some products where we re-fed the same stream to them days or weeks later turned in dramatically different scores, showing how important spam updates can be for products that depend on signature-based technology. Each product was connected to the Internet, and got signature and software updates as often as the vendor recommended.

Our goal with the test was to get between 10,000 and 30,000 messages over a one-month period. However, we discovered vendor irregularities in the middle of the test, and stopped the mail stream after two weeks, and approximately 11,000 messages.

We then went through every single message, classifying it as spam, not spam or unknown. We defined 8,027 messages as spam, for which there was no conceivable business or personal relationship between the sender and receiver, and which was obviously bulk in nature. In the not spam category, 2,386 messages that may or may not have been solicited, but which had a clear business or personal relationship between sender and receiver, or was obviously a one-to-one message, even if unsolicited and unwanted. All mailing lists that had legitimate subscriptions were considered not spam. We didn't make any mailing list changes during the term of the test.

In the unknown category were several messages that either were the result of virus double-bounces, messages that we couldn't put into one category or the other definitively, and some messages that were so malformed that we didn't know whether they were spam, viruses or just software acting up. We also took messages with duplicate message IDs and deleted them from the data set. In theory, a duplicate message ID is impossible, but spammers don't follow the RFCs, and we found more than 100 of those.