What is a false positive?

With spam, suddenly everyone cares about statistics. For the first time, system administrators are buying software that openly admits that it doesn't work all the time. Not only that, the percentages are pretty dismal. Would you buy a firewall that claims to work only 99% of the time? Or a compiler that advertises that it mis-compiles programs once every 1,000 times?

Of course, we know that with many software packages, there are going to be errors and that it won't work 100% of the time. We just don't base our buying decision on that percentage. Virus scanners don't work 100% of the time, but you don't pick a virus scanner based on published results of how often it fails.

But that's the way we buy anti-spam products, and will continue to do so for at least the next few years, with spam-catch rate and error rate as all-important statistics in the buying process. At least that's what readers tell us. One thing we found in our test this year is that all products are not alike. Several vendors called us, claiming the opposite, and would prefer people evaluate their products based on all the other features they've worked so hard to include. That's nice, but until anti-spam products work as well as anti-virus products - and they don't - we will still test for accuracy.

If you consider that numbers are the single most important part of your buying decision, you should probably know what they mean. Since most of us forgot everything we knew about statistics a few hours after the final exam in college, we present this little reminder primer. Don't worry, there's no quiz at the end of the article.

The terms false positive and false negative (along with true positive and true negative) come to us from the world of diagnostic tests. An anti-spam product is like a pregnancy test - it eventually comes down to yes or no. False positive means the test said the message was spam, when in reality it wasn't. A false negative means that the test said a message was not spam, when in reality it was.