Spam spotting: Man vs Computer
Most of us fancy our chances against the spam spewing zombie bots, but even the smartest brains have their blind spots.
Spam accounts for around 95 per cent of all business email sent by most estimations, so the occasional piece of junk mail is bound to slip by the filters. More often than not, the message seems to be so obviously spam that users can only conclude that they could do a better job than the technology; but, in truth, the human mind is too susceptible to outside influences to ever beat the machine.
To demonstrate the issues involved security software firm SoftScan conducted an in-house experiment to see who was better at detecting spam - a team of volunteers from the sales department, or the company's spam filters. It was soon apparent, despite believing that they would easily beat the machine, that the team were much slower and made several wrong decisions.
This is because despite the fact that humans run the best simulation software available, and are able to reach solutions to complicated situations in milliseconds, even without knowing all the facts. Their brains add too many variables - and not enough rules - to identify spam correctly. Not only that, brains are surprisingly easy to trick.
Most peoples' mental make-up is set to believe the facts they are given, if they seem reasonable. This is handy if you are in a boat and someone warns you not to dip your toes in the water because the fish bite. But the brain's ability to reason at mega-fast speed is also its downfall. If it has no previous experience that might tell it that the facts are not all that they seem, then the gullible brain has no reason to believe differently.
That is why phishing emails were successful when they started. A helpful email from your bank alerting you to changes that might affect you was easily believed; after all, your bank had always been helpful in the real world - what was so different in the online version? Now, most people know differently. They've learned to be suspicious of such emails, but still frequently can't resist opening up something that might be relevant to a current news event or their favourite celebrity - and canny spammers know this.
Other spam tricks that work well, but are less used include subscribing someone to a 'Google Alert' using the 256 characters available to sell a product, or to take the subscriber to a malicious website.The email comes from a legitimate source and depending on the text used is unlikely to be marked as spam by any filter.
So: can a spam email be spotted just by its subject line?
Aside from the obvious human insecurities that spammers exploit to encourage us to open emails with offers of finding the true love of our lives and/or improving sexual prowess, which are easy to detect, other emails purporting to come from legitimate business are not always as easy at first glance.
For instance, how many users are aware that real emails from eBay normally include the username in the subject line? Even then, if you are a potential victim of a targeted attack, it may still not be a legitimate email and there's very little the average user can do to spot it. This is why security firms always advise not to click on the link, but to open up a browser and access the site that way.
Humans also add emotive reasoning into the equation. An email offering to 'consolidate all your loans', would be considered spam by most, but to someone that had money problems, they might easily conclude that the user had subscribed to some advisory service. Particularly if the 'from' address seemed to confirm their suspicions.
This highlights another issue. Humans only look at the headers, subject line, and possibly a section of the main body of the text. As spam messages become more sophisticated, this will make them harder for a human to detect. In fact, a large proportion of spam today is eradicated before it even gets to the main filters, and without either a machine or a person looking at the message.
But spammers rely on the way the human brain processes information to trick it further, such as strange spellings that the brain automatically translates for the user into the real meaning. Although this technique no longer fools a computer, the user is still susceptible and frequently does not notice minor changes.
For a while this ability to decipher the spammers' message without a second thought did make humans better at spotting image spam. A pink background with blue specks and white text made it very easy for the human brain to see that the message was spam, but since the introduction of sophisticated OCR (optical character recognition) technologies into most anti-spam solutions, this trend has slowed considerably.
In contrast to the human brain, anti-spam solutions use a set of rules that allows no deviation or emotive response when making a decision. 'Computer says no' isn't just a catch phrase from TV's 'Little Britain' that reflects on the user's inability to question anything; it also demonstrates the machine's inability to come up with anything other than a positive or negative answer.
When stopping spam that is a good thing. A large proportion of junk mail relies on social engineering tactics to fool the user into thinking it is a legitimate email, and carrying out specific tasks to their detriment. Computers do not consider social engineering tactics, and therefore a major influential element is immediately eliminated from the equation of deception.
The anti-spam race
Arriving at the answer of 'is it ham or spam', however, uses a complex combination of technologies that play different roles in detecting spam. Over the years, many have come up with varying ideas, from Sender ID and SPF, to challenge response. The question also remains of how valid these technologies will be, as spam becomes more prominent in other areas of electronic communication, such as instant messaging.
In the early days, introducing new anti-spam technology could mean an increase of up to 30 per cent in detection rates and have a significant impact. Now even just a 0.5 per cent increase is praised as revolutionary. To understand why such a small percentage is so great, one needs to understand that reputable anti-spam solutions are already catching around 99 per cent of spam aimed at corporate networks.
Alas, users rarely see the 1 per cent as a remarkable feat in anti-spam technology, particularly if the 1 per cent actually equates to five or six spam messages a day. Reputation filters, introduced in one form or another by anti-spam vendors about 18 months ago, have helped to reduce this perception by eliminating the vast majority of spam messages before they reach the filters.
Reputation filters work by detecting servers that are distributing spam and automatically block traffic from them. Each time an email is accepted the filter performs an assessment. It looks-up the IP address in a number of databases that collect data about the senders of spam and viruses. Once an IP address has been identified as responsible for sending spam, messages from it are blocked before they are sent to the spam filter, and before conventional blacklists can update. Large quantities of spam can be removed in this way, reducing the load on the scanning system and minimising the effects of other new techniques.
The introduction of reputation filters was probably the last major update in email anti-spam measures. Most of the new developments carried-out in 2008 are improvements and fine-tuning on existing technologies. When the majority of spam is already being detected, there is little need to develop radically new technologies to combat email spam. The issue of how to eradicate it permanently is still important, but already we can see that spammers are turning their attention to methods other than email. The next leaps in technology are more likely to be focused on new spam distribution methods such as instant messaging and Web-based spam.
Although it is highly unlikely that spammers will lose focus on email as their main distribution method any time soon, it does emphasise the point that while technology will enable businesses to eliminate the majority of spam from their networks, it is not enough to make the problem go away permanently. Other factors such as government intervention, user education, and harsher sentencing to those caught must come into the equation.