Getting a purchase on Christmas

If you trust the postal system enough to shop online this year, the chances are that several of the places you go to buy presents will have multiple recommendations on what else to add to your shopping trolley.

According to a report issued by analyst Forrester Research last year, at least 30 per cent of larger Internet retailers - etailers - employ some sort of system to recommend related products to online customers. Indeed, this feature of online selling has proved to be a highly effective way of boosting the number of items that get added to virtual shopping trolleys as presents lists are ordered, or inspiration is sought for gift for those 'difficult' recipients.

However, although there is some smart programming involved in these systems, there are varying levels of sophistication in the underlying technologies that drives these recommenders. For instance, Pontus Kristiansson, CEO and co-founder of Avail Intelligence, reckons this number includes manual systems where the retailers have hand-picked products to show to customers; automated systems that analyse and respond to consumer behaviour are a little more expensive to implement, and therefore rarer.

Avail itself focuses on the European market but Kristiansson believes the proportion of medium-sized to large retailers who use this kind of software is closer to 15 to 20 per cent. "Having said that, interest has picked up during 2009. I have been speculating whether this is caused by the recession or not," Kristiansson says. "In 2003, when we came out of the dot-com bust, there was an incredible race among retailers for market share. They were fighting to get traffic to their sites, and were not too bothered about the profitability. It has been incredibly expensive for them to buy customers - but now the recession has forced them to look at the profitability of their traffic."

Sales statistics

Retailers have come to believe recommendation engines work. Most quote sales increases on the order of 10 per cent. Chris Mann, senior product manager at CoreMetrics, says between 5 and 15 per cent of total webstore revenue can be driven by product recommendations.

"This is a statistical game," agrees Kristiansson. "It is not black and white. It depends on which vertical market you are in; and it depends on how you deploy the recommendations. There are a number of places around a site where you can deploy, and with different amounts of prominence."

Recommendations help retailers in two ways: conversion rate, and order value. Conversion rate measures how many customers visit a site and come away having bought something, whether it was what they came to the site to buy or not. Order value is driven mostly by buying multiple products and accessories - cross-selling - or by being convinced to take a more expensive alternative to an original choice - known as 'upselling'.

According to Jack Jia, CEO of Baynote, getting users to the site is a major problem but, even when they have turned up, many leave without success. The snag is that etailers have to rely on prospective buyers' ability to express the nature of their desire, and interrogate the correct classified site listings and/or employ the correct search terms. "Eighty-three per cent of people can't find what they are looking for online," Jia claims, "even if there is a website that has the perfect product for them."

Preemptive selling

If conversion is a big problem, then it is important to have recommendations waiting for the potential customer the minute they roll up. Even if someone has not shopped at a site before - and therefore has no record in the form of browser cookies - you can garner hints of what the user is looking by examining the referral string that the browser provides when it is directed from a search engine or from another page. London furniture store Heals, for example, is one retailer that uses this technique.

However, if order value is a higher priority, then it is better to deploy recommendations closer to the point where the customer is buying something, although it is possible to go too far, and send users running to an alternative outlet.

Very often, consumers are enticed to visit a store because they see a good deal on a popular item. The problem that faces many implementers of automated recommendations is that it is all too easy to recommend other very popular items.

Says Avail Intelligence's Kristiansson: "People often look at the order history of a shopper. They then take a particular item, and look at how frequently other items are bought together with it. That becomes a very complicated way of producing a top-seller list. Britney Spears will be at the top of that list, no matter what."

Popular items make poor recommendations on two levels. For retailers they attract low margins because they are often loss-leader products to get customers 'through the front door'. "You want the recommendations to be relevant, but surprising," Kristiansson avers. "and you have to look a long way down the long tail to do that."

"What's really controversial is showing cross-sell products in the cart itself," says Linda Bustos, e-commerce analyst at Elastic Path Software, which sells webstore software and services to retailers. She stresses that testing these techniques is vital before introducing them.

Pop-ups pop up

Pop-ups tend to be pet hates for online shoppers and another potential troublespot is the fly-out window. "We are starting to see with some creative sites things where, when you are hovering over a menu item then the recommendation flies out," Bustos says.

This type of idea, which may sound great in an internal company meeting, can deter users. "But that is something that can be tested," says Bustos, to check whether it actually does more harm than good. "Generally, anything that involves a pop-up is going to be controversial."

Labelling is more subtle, but important. Testing across many sites has revealed that vague wording such as 'You might also like' puts people off compared to descriptions of recommendations that are closer to the truth, such as 'customers who bought this also bought these'. This is an effect that reaches its logical conclusion with the adoption of user rating and recommendations by a growing number of retailers (see 'Favourite things', p56).

"It does not have to be customers' choices, it can include staff picks," Bustos observes, citing DVD and book stores as places where people will trust staff recommendations more. "You trust the staff because of the stereotype of the guy who works at the movie store, who does nothing but watch movies."

The shop also gets a benefit from recommending less popular items because they tend to carry higher margins; and it is possible to tweak the engine to pull out more profitable products. According to Baynote's Jack Jia: "Harry Potter books don't need recommendations - you don't want to recommend them either, because they carry no margin. You want to recommend long-tail items."

Category tagging

Vendors such as Avail and Baynote argue that their algorithms are designed to avoid simply building most-popular lists in the way that slightly older techniques such as collaborative filtering (which Amazon said it used in a rare 2003 technology briefing paper) seem to do. However, systems generally need some tuning to stop them from recommending the inappropriately, although etail giant Amazon courted controversy when it tagged books that tackled issues such as homosexuality as 'adult' titles. For other products, it is much more clear cut: "Never, ever recommend items from the adult toy catalogue," advises Avail Intelligence's Kristiansson.

An ongoing problem for all recommendation systems is the compute power needed. The aforementioned Amazon briefing paper described how the webstore could not apply techniques that would provide the most relevant recommendations because they would not scale to millions of users, although things may have changed since that statement was formulated. Some of the most compute-intensive tasks can be performed offline in batch runs, particularly for item-to-item comparisons.

Kristiansson argues batch calculation does not work well for situations where you want to take the user's context into account, which means being prepared to use more servers: "There is a conflict between relevancy and performance. To get good recommendations, there is a risk that your requirement on hardware will grow exponentially with the growth of the underlying dataset."

Related forum discussions

	To start a discussion topic about this article, please log in or register.