How ‘Big Data’ is helping firms understand the world

One of the top trends for many businesses in the coming 12 months is likely to be improving their insight into the activities of customers and the wider marketplace.

This is because it is no longer good enough for firms to have strong reaction times in order to keep up with the latest developments in their industry. Increasingly, the most successful organizations will be those that are able to spot trends before they become obvious and accurately predict where their company needs to be.

While this may seem like a highly complex and costly business, the information needed to make these decisions is already out there and is growing all the time. It was noted recently by Newsweek that the average person’s current data footprint amounts to almost one terabyte. The publication stated this is enough information to store eight trillion yes or no questions.

“Today’s big data is just the tip of the iceberg,” it observed. “The total data footprint of Homo sapiens is doubling every two years, as data storage technology improves, bandwidth increases, and our lives gradually migrate onto the internet.”

But taking full advantage of this will require investment in big data analytics. International Data Corporation recently forecast that this market will increase at an compound annual growth rate of more than 27 per cent until 2017, so it is clear that this message is getting through to IT leaders and other senior executives.

But what will this lead to in real terms? Newsweek noted that there have been many examples recently of big data-led projects that have led to crucial insight – and these have taken place across all sectors. However, in many of these, observers have not appreciated how they use key big data technologies to ensure success.

“Big Data will change the way we look at everything,” said Michael Hiskey, Vice President and Head Product Evangelist for Kognitio, a provider of analytical software for big data applications. “The implications for business, research, politics, and so many other endeavors are only beginning to be understood.  There is so much out there, and 2014 will see more of it uncovered than has been seen over the past three years combined.”

For instance, Stanford economist Jon Levin teamed up with eBay to analyze how prices are established in real-world markets, taking advantage of the fact that many eBay merchants perform small-scale experiments to determine the best price to charge for their products.

Newsweek said: “By studying hundreds of thousands of such pricing experiments at once, Levin and his co-workers shed a great deal of light on the theory of prices, a well-developed but largely theoretical subfield of economics.”

While it showed that many existing ideas about this theory were largely correct, it also highlighted several key errors in thinking. The research has proven highly influential to vendors when making pricing decisions, but would not have been possible without the power of big data.

Other sectors that have benefited from predictive analytics driven by studying very large data sets include infrastructure, which can use cell phone tracking to identify patterns in people’s movement and help to ease congestion, while in healthcare, information can be used to spot outbreak of illnesses such as flu before they become widespread – allowing organizations to cope much better with epidemics.

Its impact has even been felt in politics, with Newsweek highlighting the analysing of polling data made by Nate Silver. In the 2008 and 2012 presidential elections, Mr Silver became one of the country’s best-known statisticians with his analytical model, that studied a wide range of polls to forecast the results. In 2008, his FiveThirtyEight website successfully predicted the outcome of 49 out of 50 states, while in 2012, all 50 results were forecast correctly.

Newsweek said: “The list goes on and on. Using big data, the researchers of today are doing experiments that their forebears could not have dreamed of.”