Big data ‘to revolutionize’ decision-making

While the age of big data is set to have an impact across many levels of a company, one of the most disruptive changes will be how the technology impacts decision-making within an organization – particularly when it comes to affecting how much influence senior personnel will have in the process.

In a keynote speech at a conference in New York last week, principal research scientist at MIT's Center for Digital Business Andrew McAfee said that the single biggest factor created by data analytics will be to challenge the dominance in many businesses of what he terms the 'HiPPO' – the highest paid person's opinion.

For many businesses, the usual way of making decisions is to rely heavily on the expertise and knowledge of these individuals, particularly when relevant, reliable data is hard to come by. In these situations, gut feelings remain the most commonly-used decision-making tools.

But this is changing as more firms invest in big data solutions that provide them with access to information that would previously have been unavailable, or would have taken too long to effectively analyze. In the new era, Mr McAfee noted a conflict is arising between HiPPOs who still rely on intuition to make decision and 'geeks' who apply quantitative figures in the form of big data to the problems. When the success of these strategies is compared, the results are clear, Mr McAfee said – "the geeks win".

He added that it is not just the volume of data that is leading to a major shift in the way firms make decisions, but also the variety of it and the speed at which companies can gather and process it. He said: "The social web is throwing [up] weird kinds of data, much more ragged than we're used to."

Technologies such as mobile phones and tablets are adding to this, as "they leave behind a sensor exhaust trail" that offers much more detailed information on individual users than ever before.

Mr McAfee also offered several case studies to illustrate how big data is changing the way organizations from all sectors of the economy do business. For instance, in aviation, information gathered by an aerospace services firm offered insight into the landing patterns of commercial jets. This information could then be sold to airlines, who could apply it to help cut airport wait times, improving the customer experience.

One example of how the new, quantitative approach that big data offers can improve on gut feelings and the opinions of HiPPOs was also seen in the wine business. Back in 1989 Princeton University economist Orley Ashenfelter suggested that weather data could be used to predict the quality of wine before it was even sampled.

This outraged wine connoisseur Robert Parker, who described the claims as "ludicrous and absurd". However, Mr Ashenfelter made a case for his statement, eventually presenting a paper that highlighted how wine quality correlates with key meteorological metrics – vindicating the data-driven approach to assessing vintages.

But Mr McAfee warned that making a success of big data analytics will involve ensuring all players are on the same page and avoiding a battle for control between the old world and the new. "Speak the language of the other side. Don't be part of this broken dialogue between the geek side of the house and the suits," he said.