Baseball batsman from the Boston Red Sox

Thomas Davenport: 'Big Data at Work' interview

Big data is trending as the pass-code that will unlock the secrets of commercial success. But what is big data, and how does it differ from conventional business analytics? Thomas Davenport's latest book explains all.

Despite the first two words of his new book being "big data", Thomas Davenport isn't exactly an enthusiast for the term. "There is a lot of confusion about what it means," he says, before describing it as a relative term, where the epithet "big" should not simply be restricted simply to the volume of data.

"I describe big data as data that is too big to be processed on one server, too fast-moving to be sequestered in a data warehouse, or too unstructured to fit into a conventional database." He explains that this lack of structure "gives organisations the most problems. For that reason and others, I am not a fan of the term. But we seem to be stuck with it for the moment".

Davenport is careful to point out that his new book – 'Big Data at Work' – isn't just another book about business analytics ("I should know, because I wrote several popular books on that subject"). At first he thought it might be, but as his research brought him back time and again to the lack of structure in the data, the difference started to become clear. "It requires new technologies and methods to process big data. Once you have done that, the analytical approaches are similar to what we have used in the past.

"Also, the fast-moving nature of big data means that we also have to develop new approaches to continuous decision-making and action. Some organisations are beginning to do that. I call it 'Analytics 3.0'. Traditional business analytics could be much slower and episodic."

According to Davenport, as the volume and types of data continue to expand there will be many opportunities to better understand customers and businesses. "In the past, the organisations that have undertaken aggressive efforts to capture and analyse information have done well relative to their competitors. I see no reason why this won't continue or accelerate, and big data is just the latest incarnation of this trend."

The message is clear. Big data is here to stay and now that we have a method of analysing and drawing conclusions from it, those in the know will be the winners, while those who prefer to do things the old-fashioned way will be left behind. Surely there's something wrong with that argument? After all, there's always the nagging temptation to think of management as being as much an art as a science, where intuition and experience count for as much as data. Davenport reluctantly acknowledges the point, but isn't really buying this.

"Across almost every field of human endeavour, research suggests that data and analysis yield more accurate and reliable decisions. There are, of course, situations in which you can't get data, and we have to fall back on intuition. And intuition can be a fairly accurate guide to decisions if you have a lot of experience with that decision, and you have closely monitored your errors. For the most part, however, intuition and experience should be the last resort for a decision rather than the first one."

World of sport

Perhaps one of the most interesting areas where data analysis is taking off is in sport. It's virtually impossible to watch a rugby union game without seeing banks of data analysts at their laptops monitoring possession, territory, handling errors, passes completed and myriad other factors. This is an application that really excites Davenport, who is currently working on a new study on sports analytics.

He says that in almost all sports, "analytics are a means of improving performance. They typically provide only a small edge, and there is a lot of chance that determines the outcome of individual games and so it wouldn't be accurate to blame analytics or analysts for a loss in a single match".

And yet back in the author's hometown of Boston, the Red Sox baseball team seems to have enjoyed something of a renaissance after adopting analytics. Davenport describes how in the past decade the team has undergone a conspicuous upturn in fortune, winning three World Series championships, an achievement they had failed to deliver for 85 years. "They have obviously done a few other things than adopting analytics. But that is probably the most sweeping change. In football, the English Premier League is beginning to embrace data and analytics with a similar level of seriousness. I hear that the Welsh Rugby Union is doing great things as well."

Davenport is a sociologist by academic background who calls himself a "sociologist of information. I look at how people and organisations make effective use of information. I wrote several successful books on how to compete with business analytics, and my big data book is an extension of that research which I wrote when I realised that my previous books were not sufficient".

He says that if there is one moment in 'Big Data at Work' that is worth the cover price, it is his assertion that the job title 'data scientist' is gaining in popularity, becoming "the sexiest job of the 21st century". But there are drawbacks and Davenport confesses to being unimpressed "by many of the social media sentiment analysis applications. Understanding what your customers are saying and thinking about your company and its offerings is a useful thing, but interpreting human language is difficult. The other problem is that sentiment varies between levels of positive and negative. I rarely see companies taking action on the results. I know of a few that had pilot projects in this area, but abandoned them because it was either too difficult, or the lack of action taken didn't warrant the expense of the analysis".'

Then there is the question of ethics. Despite the fact that we need to ensure that we are not breaking the law, it is "often difficult to know just how much customers actually care. They say that they care a lot about their privacy, but they regularly reveal private information over social media and give away personal data in exchange for small gifts and discounts".

'Big Data at Work' by Thomas Davenport is published by Harvard Business Review Press, £20

Recent articles

Info Message

Our sites use cookies to support some functionality, and to collect anonymous user data.

Learn more about IET cookies and how to control them

Close