How we did it

We evaluated enterprise-level anti-spam products by installing them in a production e-mail environment for approximately one month. Most products were installed on servers at our test lab; the four off-site anti-spam services were tested using a 24M bit/sec Internet connection. To accommodate simultaneous testing, a dual-processor Intel server with 1.4-GHz CPUs and 3G-bytes of memory was used with VMWare's GSX server to create a different virtual machine for each product. Two products (Tumbleweed and Corvigo) ship on appliances and came with their own servers; two others (MailFrontier and SurfControl) ran on other systems because deadlines did not allow us the time to install them on our own hardware.

In the first part of the testing, we evaluated how each product does one important job: filtering spam. We took the tester's Opus One incoming mail stream and simultaneously re-fed it to each of the test systems in very close to real time. This meant that each product was seeing the same spam more-or-less simultaneously, and, more importantly, was seeing it as it flowed into our network from the Internet. Sending canned (old) spam would have been a lot easier, but would have been a poor test because it's a lot easier to filter out old spam than new spam. Each product was connected to the Internet and got spam signature updates as often as the vendor recommended.

Our goal was to get approximately 10,000 messages for a good statistical sampling, so we let this test run for most of June. Our actual total was more than 12,000 messages, which we reduced to 11,324 messages to fall on midnight boundaries.

We then looked at every, single one of those messages and divided them into one of three categories: spam, not spam, and don't know. We defined as spam the 7,840 messages for which there was no conceivable business or personal relationship between sender and receiver, and which was obviously bulk in nature. In the not spam category were 3,648 mail messages which may or may not have been solicited - they either had a clear business or personal relationship between sender and receiver, or was obviously a one-to-one message, even if unsolicited and unwanted. All mailing lists that had legitimate subscriptions were considered not spam. We didn't make any mailing list changes during the test duration.