Thursday, 30 May 2013

"Users of email will not put up with it"

If you have ever used email, then you will know about spam, or unsolicited bulk email.

You may also know that various intelligent and technical people have proposed solutions to deal with it: solutions that they invariably believe to be clever and novel. However, the solutions usually turn out to have some practical flaw which makes them unworkable.

At some point, "final ultimate solutions to the spam problem" became such a frequent occurence within certain sorts of newsgroup and certain sort of forum that a form letter named "spamsolutions.txt" was created as a generic response. It begins like this:
Your post advocates a
( ) technical ( ) legislative ( ) market-based ( ) vigilante
approach to fighting spam. Your idea will not work. Here's why...
A decade ago, you could not even discuss solutions to spam in a public forum without getting "spamsolutions.txt" in response. It would be posted by an unbearably smug person who believes that merely knowing about "spamsolutions.txt" makes them superior to you. There is a spam equivalent of "Godwin's Law" here.

It is therefore ironic that "spamsolutions.txt" has been thoroughly debunked by events during the last decade. Namely, the mass migration of email users from the old peer-to-peer email system to centralised email systems, particularly Gmail.

Suppose it is the year 2001, and you join a forum such as Slashdot. A discussion about spam comes up, and as a naive new user, you post the following message:
My idea for fighting spam is that we should all sign up for webmail services from large email service providers. Like Hotmail or Yahoo. Or Google, if they launch an email service some day. Then the service providers can build an ecosystem in which (1) information about spam classifications is shared widely, and (2) emails from outside the ecosystem are not automatically trusted.
Oh dear! You immediately get several replies, all of which look like this:
Haahaha LOL @ the noob.
Your post advocates a
(X) technical ( ) legislative (X) market-based ( ) vigilante
approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
(X) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
(X) It is defenseless against brute force attacks
( ) It will stop spam for two weeks and then we'll be stuck with it
(X) Users of email will not put up with it
(X) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
( ) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
(X) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
(X) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
(X) Huge existing software investment in SMTP
(X) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
(X) Armies of worm riddled broadband-connected Windows boxes
(X) Eternal arms race involved in all filtering approaches
(X) Extreme profitability of spam
( ) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
(X) Bandwidth costs that are unaffected by client filtering
(X) Outlook

and the following philosophical objections may also apply:

(X) Ideas similar to yours are easy to come up with, yet none have ever been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
(X) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
(X) Why should we have to trust you and your servers?
(X) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
(X) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough

Furthermore, this is what I think about you:

(X) Sorry dude, but I don't think it would work.
(X) This is a stupid idea, and you're a stupid person for suggesting it.
(X) Nice try! I'm going to find out where you live and burn your house down!
In true Slashdot fashion, these formulaic replies are quickly moderated up, while your misfit message sinks in shame.

Within a decade, you will have been proved right. "spamsolutions.txt" will rarely be seen, and almost everyone who posted their smug replies will have Gmail accounts.

It is not that they were wrong about the disadvantages. All of them are valid objections. But it turns out that they are not as important as it might have seemed. Users of email will, in fact, put up with it. No... they'd prefer it!

While attacking the original poster's naive ideas about spam, the "spamsolutions.txt" response makes equally naive assumptions about users, technology and business. (If you find this amusing, then you may also enjoy searching vintage Slashdot for optimistic predictions about the Year of the Linux Desktop.)

Spam is still being sent, in large quantities. I still get dozens of spam emails in a day, many from fake academic conferences that will accept literally anything from a paying customer. But I have to go specially to my Spam folder to read any of them, because Gmail classifies nearly all of them correctly, and they disappear unread. The classifier works really well because it shares information with other users. Whenever a user marks a message as Spam, or Not Spam, the system has learned something about all of the messages that look a bit like it. There was no equivalent on the old system: no possible way to obtaining such rich classification data from masses of users.

Information sharing was attempted on the old peer-to-peer system. Blacklists of known spam sources, blacklists of phrases likely to appear in spam, that sort of thing. It never worked very well. The centralised system does all of it better: better information sharing, and a better quality of information to share.

I held out as a user of the old system for many, many years. I did not stop using it because of spam - no, it was anti-spam blacklists that finished it. I found that my messages would sometimes just vanish. They were being wrongly classed as spam, somewhere en route, and I would never know where because of the huge number of different systems involved. Who knows which blacklist I had fallen foul of? Which "sendmail" rule was dropping my emails? I will never know, and do not much care, because the solution was to move to a centralised email service that nobody would dare to blacklist. If your mail server drops everything from "gmail.com", then that is your problem, not mine. And that's how it has to be.