The dangers of flawed data

As businesses come to rely more heavily than ever on data to inform their decision-making through all parts of their organisation, making sure this information can be trusted is vital.

In the past, if a certain piece of data turned out to be inaccurate, the consequences would be limited. But in today's environment, when firms are much more interconnected and dependent upon data for long-term decision-making, the ripple effect can be serious.

This is an issue recently highlighted by Thomas Redman, PhD, author of Data Driven: Profiting from Your Most Important Business Asset. In a piece for the Harvard Business Review, he wrote that the problems are particularly acute when the C-suite is relying on this data for high-level strategising.

He said: "An error in a customer database only fouls up a transaction or two. But aggregated bad data can send a decision awry and hurt the company for years."

At the moment, many managers accept that with the explosion in assets caused by the big data era, these problems are set to get worse, not better. But too many of them continue to incorrectly believe these issues will not affect their planning. Therefore, he noted it is past time for businesses to stop making excuses for flawed data.

Dr Redman said that recognising the issue is the first step, and he highlighted four key areas that businesses need to focus on in order to crack down on the issue of poor data.

"Data quality issues come in many forms – from not having the data you really need, to data that are easy to misinterpret, to data that simply can't be trusted", he said. "Worst are the data quality problems that you don't even know you have. Unfortunately, they are all too common."

The first problem area is firms not having the right data, which means key markers are missed and executives do not have a complete picture. Therefore, understanding what questions firms need to ask – and what resources will provide the right answers – is a critical first step to improving data quality.

However, even if the right information is available, misinterpreting it is another common issue. For instance, Dr Redman noted one classic result of this is asking 'how many customers do we have?', without being clear on how their firm is defining a customer.

The next questions organisations need to ask is if they can trust the data they are receiving – and if so, why that trust is justified. This can cause issues when managers do not have confidence they data is accurate, which means they dismiss it from their decision-making processes.

"It's particularly disheartening when a big data analysis yields a stunning new insight. But the big data team spent 90 per cent of its time cleaning up the data," Dr Redman said. "You know they did the best they could, but you're uncertain they got enough of the errors."

This can render a big data analytics programme irrelevant, as if businesses cannot trust the data going into the analysis, they cannot trust the insight that comes out. 

Dr Redman therefore concluded that it's time for managers to recognise that simply doing their best in spite of poor data is no longer good enough. He said: "Decision making is just so much more effective when you have a complete, trusted, fully-understood picture."