Applications /

Weighing the true costs of spam filters

By Paul McNamara
Network World, 07/29/02

Depending on your point of view, spam filters are:

A) Necessary.

B) Evil.

C) A necessary evil.

And the three might not be mutually exclusive.

Geoff Duncan is technical editor, subscriptions manager and Web developer for TidBITS, an e-mail subscription and online publication that caters to the Macintosh crowd. He recently wrote about the results of his informal research into the effects that spam filtering by others is having on his organization's ability to electronically deliver TidBITS. (You can read his complete report at db.tidbits.com/getbits.acgi?tbart=06866.)

"In the last year or so, we've noticed a new trend: Some weeks, we get errors from hundreds [or even thousands] of subscribers whose servers refuse delivery of TidBITS issues," Duncan writes. "On the heels of these errors, we usually receive a flurry of complaints: 'Why didn't I get this week's issue?' or 'Please fix my subscription - I didn't get TidBITS today but your system says I'm still on the list.'"

The problem is false positives. Ham-handed filters are erroneously flagging TidBITS content as being spam - or in some cases, a virus.

While few will dispute that spam is to e-mail what grubs are to lawns, there might need to be more discussion about whether currently available spam filters and filtering techniques constitute an appropriate pesticide or a modern-day Agent Orange. Although he uses filters himself, Duncan has his doubts.

"In short, we're starting to see signs that e-mail, often hailed as the Internet's killer app, is in danger of becoming an unreliable, arbitrarily censored medium" he says.

In one week alone, a whopping 10% of TidBITS recipients - 4,000 readers - failed to receive the news they had requested simply because a TidBITS writer made a passing reference to Viagra. Worse yet, in writing about the filtering fallout Duncan was reduced to referring to Viagra as "a well-known Pfizer drug for men," lest repeating the brand name cause his report to again run afoul of the filters.

Self-censorship has apparently become a way of life at TidBITS. And while simply avoiding the word Viagra might seem a small price to pay in the interest of spam control, it quickly becomes apparent that writing around these filters is easier said than done. Other examples from Duncan's file:

Mentioning Napster in an article got TidBITS rejected by about 120 e-mail servers. (Duncan had to call the offensive proper name "a well-known peer-to-peer music swapping service.")

Another story was blocked by more than 1,100 sites for mentioning a particularly annoying online ad campaign for video cameras. (Yes, it was X10, and no, Duncan could not risk printing the name in his piece about filtering.) Other sites rejected this story because it included the word "undress."

The words "my" and "pictures" in succession kept another batch of TidBITS material from hundreds of intended recipients.

While the benefits of spam filtering are clear and clearly desirable, Duncan believes that not enough emphasis is being placed on the costs.

"As much as on-target filtering might save administrators and users time, money and trouble, filtering that backfires also has direct costs," Duncan says. "Part of that cost is passed off to the sender whose e-mail has been improperly identified: Every time spam filtering hits TidBITS, I get to track the problem down, deal with e-mail administrators and assuage irritated subscribers."

Those doing the filtering also pay, he says, in administrator time and the lost productivity - if not actual lost business - that can come with missing e-mail.

Something tells me this issue isn't going away any time soon.

Yes, we use spam filters at Network World, but that shouldn't stop you from writing. The address is buzz@nww.com.