What is a false positive?

With spam, suddenly everyone cares about statistics. For the first time, system administrators are buying software that openly admits that it doesn't work all the time. Not only that, the percentages are pretty dismal. Would you buy a firewall that claims to work only 99% of the time? Or a compiler that advertises that it mis-compiles programs once every 1,000 times?

Of course, we know that with many software packages, there are going to be errors and that it won't work 100% of the time. We just don't base our buying decision on that percentage. Virus scanners don't work 100% of the time, but you don't pick a virus scanner based on published results of how often it fails.

But that's the way we buy anti-spam products, and will continue to do so for at least the next few years, with spam-catch rate and error rate as all-important statistics in the buying process. At least that's what readers tell us. One thing we found in our test this year is that all products are not alike. Several vendors called us, claiming the opposite, and would prefer people evaluate their products based on all the other features they've worked so hard to include. That's nice, but until anti-spam products work as well as anti-virus products - and they don't - we will still test for accuracy.

If you consider that numbers are the single most important part of your buying decision, you should probably know what they mean. Since most of us forgot everything we knew about statistics a few hours after the final exam in college, we present this little reminder primer. Don't worry, there's no quiz at the end of the article.

The terms false positive and false negative (along with true positive and true negative) come to us from the world of diagnostic tests. An anti-spam product is like a pregnancy test - it eventually comes down to yes or no. False positive means the test said the message was spam, when in reality it wasn't. A false negative means that the test said a message was not spam, when in reality it was.

We often think in terms of error rates, but with many diagnostic tests the kind of error is a big deal. It's not enough to know that the test is wrong 29% of the time. We want to know what kind of wrong. Spam tests are exactly like that. A false positive means that good mail might have gotten lost, while a false negative is just annoying. We care more about false positives than we do about false negatives (unless the CEO is getting inundated with false negatives). In addition to wanting to know how many errors there are, we also want to know what type they are.

You also may want to adjust the behavior of the system, so we gave points to products that let you change its behavior. Based on your tolerance for false negatives (spam in the mailbox) or false positives (mail mis-marked as spam, lost or delayed), you may want to set the product to have different thresholds. In our test, we didn't expose those thresholds. Instead, we asked the vendors to pick thresholds and to tune their products such that the false-positive rate would be kept to less than 1% of all e-mail.