Two weeks ago, a few IT journalists quoted a report that painted challenge/response (C/R) spam filtering in glowing colors. But as we've said before, C/R is a bad idea, which causes false positives, poor deliverability, and turns its users into spammers.
C/R is by no means the final, ultimate solution to the spam problem. So how come this report is so much at odds with the views at Ferris Research?
The press was quoting a survey, conducted by a new analyst company on the scene. Users of spam control were asked to rate the spam filtering products they used. It seems the users rated C/R more highly than any other type. However, after studying the survey methodology, we conclude that it's heavily biased toward C/R -- although we don't know whether this bias is by accident or design.
The survey questions calculate an aggregate score for a user's spam filter, based on measures such as the number of spam messages that make it through to the inbox -- false negatives -- and the number of legitimate messages that get filtered out -- false positives. The problems with the measures and the way they are aggregated into a final score include:
- The number of false positives are weighted similarly to the number of false negatives -- but in the real world, a single false positive is far more significant than a false negative.
- It's practically impossible to get an accurate count of false positives by simply asking users -- and from our research, we conclude that users of C/R technology are subject to more false positives than users of conventional spam filters, either because many challenges never reach their intended recipient, or because the challenges appear untrustworthy.
- The measures take no account of the fact that C/R causes innocent third parties to receive misdirected challenges, because spammers usually forge the envelope sender of their messages -- not only is sending of such backscatter poor net citizenship, it can cause a C/R user's outbound email to be filtered as spam when their challenges hit spamtrap addresses.
Again, for more on our views on this subject, see Why C/R is Bad, and More About the Backscatter Problem.
In summary: Don't believe all that you read in the press. Or, as Mark Twain quoted Benjamin Disraeli:
- There are three kinds of lies: lies, damned lies, and statistics.
... Richi Jennings