For many businesses, the concept of big data will revolve around the gathering of large quantities of information and having an effective analysis solution such as Hadoop that can deliver fast responses to whatever queries an organisations may have.
But these technical aspects are only half the challenge. In many cases, what holds companies back from reaching their full potential is that they are not asking the right questions of their initiatives, which means they do not have the best insight into how to improve.
It was noted by Forbes contributor Howard Baldwin that this is a particular issue when companies are dealing with 'unknown unknowns' – that is to say, occasions when they themselves are not aware of what knowledge they are lacking. As the amount of data organisations have to deal with grows, this can become a serious issue as there is simply so much data available that employees are never sure what they're looking at.
One company that suffered from this was consumer goods sales and marketing service Crossmark. It employs over 30,000 people across North America, Australia and New Zealand who visit retail stores and collect huge amounts of information, from what stores have in stock to pricing and even what displays look like.
A key challenge with this amount of information is that it leads to a large amount of "noise, static and other confusion". At the same time, shifting demographics mean traditional ways of interpreting data no longer apply.
Crossmark gives the example of a store that has issued a loyalty card to a customer named Maricella. The data from this may indicate she lives in a hispanic multigenerational household.
But making assumptions about her based on buying behaviour alone can lead to the wrong conclusions being drawn. For instance, it can be hard to determine whether she is the mother or the daughter, but this will have a major impact on the insight.
"Maricella is what we call a two-hundred-percenter," explains Alexandros Siskos, Crossmark's vice-president of analytics and insights. "She's a millennial who's 100 per cent Mexican at home and 100 per cent American at school or the mall. She behaves like any other kid in that age group."
This can make it very hard to tailor messages to her directly. In this scenario, companies also have to factor in that hispanics cannot be categorised as a single monolithic group, and misunderstanding this can lead to mistakes when companies are planning personalised messaging. At best, they may deliver irrelevant messages, but in worst-case scenarios, they could end up insulting their customers.
In the past, Crossmark's solution to this was slow and painful, involving many Excel spreadsheets – some so large that they would run out of columns and rows, and would place great strain on the company's computing resources.
"We had to figure out a way to fluidly and fluently walk through all the data,” Mr Siskos said. Crossmark's big data solution helps greatly with this, and its improved speed means the company can devote far more time to testing out hypothetical scenarios and queries to identify the best results.
"These technologies are great because they cut out the wait time. It allows us to do the what-if and actually filter out the noise to get different views of the business," he continued. "Big data allows us to ask different, clarifying, probing questions. Our goal is not to be the expert, but be the best question asker."