Key reasons why big data projects fail

Even though big data analytics tools are being increasingly implemented in all types of businesses, and the technology is well and truly moving beyond the 'hype' stage of development, many firms are still set to experience disappointment if they try to go it alone with the solutions.

In a recent piece for InformationWeek, Matt Asay, vice-president of community at MongoDB, observed that more companies are investing time and money into big data in the hope of reducing customer churn, improving their user experience or analysing financial risk, among other goals.

But he noted one of the biggest issues in the industry at the moment is that "most companies simply don't know what they're doing when it comes to big data". As a result of this, it's no surprise that firms are spending large amounts to attract skilled data scientists in order to provide the expertise they need – with the average salary for these positions now reaching $123,000.

He therefore highlighted several common problems that businesses encounter when they come to deploy big data solutions. In many of these, the root cause can often be traced back to companies making purchases in technology alone, without considering whether they have the ongoing skills and experience necessary to implement it properly.

As such, the best that firms can do it to ensure they provide their staff with the right understanding of what big data is likely to achieve. This may well involve finding a big data provider that can offer both the technical and service skills necessary for success, and will be able to advise on best practices and how to overcome challenges.

Helping customers discover what they should be asking big data analytics tools may be a key factor, as one of the key issues Mr Asay identified was that users often ask the wrong questions of their tools.

He explained: "Too many organisations hire data scientists who might be math and programming geniuses but who lack the most important component: domain knowledge." 

To prevent this issue rising, he suggested it is often best to develop the necessary analytical skills from within the organisation, as it will be easier to learn tools such as Hadoop than it will be to teach new hires about the business.

"Big data is all about asking the right questions, which is why it's so important to rely on existing employees," Mr Asay said. However he also cautioned that even with the right domain knowledge, getting off the ground with big data can be tricky, so it is to be expected that organisations will still fail to collect the right data and ask pertinent questions at the start.

What separates the most successful companies from those that never get to grips with big data will therefore be how they respond to any setbacks and the preparation they put into their schemes.

Mr Asay observed that with many big data platforms now "immediately and affordably accessible" due to technology such as cloud computing, this lowers the cost of trial and error and makes a 'start small, fail fast' approach possible.