Big data hype ‘passing its peak’

Enterprise excitement over the potential of big data may be passing its peak, and this is good for the industry as it will lead to more businesses becoming realistic about what they can achieve with the technology.

This is according to statistician Nate Silver, who was delivering a keynote address at the recent Rich Data Summit in San Francisco. Data Informed reports that he highlighted a list of misconceptions about the technology that may be now starting to fade as familiarity grows.

Mr Silver observed there has been a lot of reporting that has overstated the capabilities of big data analytics, which treat it as a magic solution that can instantly solve an organisation's problems.

"You get your data, you press a button and all of a sudden you have extremely valuable output. This idea is very wrong and dangerous," he said.

The statistician observed that interest in big data has started to peak, with more businesses looking beyond the initial hype. For instance, he observed more organisations are starting to talk about 'data science' rather than 'big data', which indicates a growing understanding of the technology.

"It's to the point that big data is being dismissed as over-hyped, and that's a good thing," he said.

Mr Silver, who came to prominence through his use of analytics to aggregate polling data in the 2008 US presidential election – correctly predicting the outcome in 49 of 50 states – said businesses must leverage their big data capabilities in the right way if they are to be successful, which requires looking beyond the initial excitement.

"Just collecting more data can get you more ways to fool yourself. Data scientists aren't interested in data for data's sake. We are interested in relationships," he said.

Businesses also need to be mindful of any human elements that may affect their results. One common error is to start with a gut feeling when championing an issue, then seeking out data that will support this assumption, ignoring any contradictory evidence. Mr Silver said the reverse approach is needed, with companies starting with an analytical mindset.