Don’t be afraid of failure with big data experiments

With the rise of big data gaining interest among businesses of all sizes and across every industry, many companies are set to get involved with the technology and implement their own solutions for taking full advantage of their information.

But many of these might be nervous about the potential impact of strategies that do not pan out, particularly if they have never used big data analytics before and are unsure of where to begin. Therefore, they may be so worried about the risk of failing that they do not even start using the tools that are available to them.

But this should not be putting companies off. With a new and rapidly-evolving technology such as big data, it is inevitable that there will be a period of trial and error and enterprises get to grips with the technology and discover what it can offer to them. However, the key for many firms is making sure that when these failures occur, they do so in such a way that the businesses can learn from them and improve their chances of success next time.

This is aided by the fact that with technologies such as data analytics tools have become much cheaper in the past few years, which means that unsuccessful projects do not come with the huge financial penalties of the past.

In a recent interview with the New York Times, Alistair Croll, entrepreneur and author of the book Lean Analytics, explained that many of the key costs that were barriers to experimentation a decade ago have been slashed or removed entirely. For instance, hardware and software availability has grown dramatically, while the advent of cloud computing has meant users can simply rent the infrastructure and applications they require rather than making large capital investments.    

"When things like that happen, companies focus less on costs and more on experimentation about what is going to make their original idea work," Mr Croll said. "There is more desire to experiment."

This will inevitably lead to more failures, but this is something that businesses need to view as a positive, he continued. He also noted that it is not just a firm's own unsuccessful projects that can be reviewed for useful insight, but those of others. Mr Croll stated forward-looking businesses collect and analyse information about the failure of others, while this information is cheaper to create and easier to come by.

What this means is that when it comes to their own operations, they are likely to make "new and interesting" mistakes, rather than simply repeating the missteps of companies that have trodden a similar path before, which the expert noted is usually a good thing.

Mr Croll added that a key result of this new environment, where failure is seen as more of an opportunity than a threat, is that it is changing organizational leadership towards a more structured business learning environment.

"In the past, a leader was someone who could get you to do stuff in the absence of information," he said. "Now it's the person who can ask the best question about what's going on and find an answer."