Is your firm getting full value from Hadoop?

As more businesses come to recognize how better insight from their data, the adoption of tools such as Hadoop to help with the analysis of these digital assets is growing all the time. But many firms may come across problems in their full rollout that mean the technology is not achieving its full potential.

Companies often find that even though they know the right data is there, they do not know how to access it, or if they do, they find the results are not useable. It was stated by Gary Nakamura, chief executive of Concurrent, in an article for AllThingsD that this means a firm has entered the "dark valley" of Hadoop.

Warning signs that an organization is in this situation include firms getting unexpectedly poor results from their Hadoop deployments. Companies may also have problems testing and deploying the applications that are supposed to extract that value, leading to delays and cost overruns in the deployment of solutions.

One quirk of Hadoop is issues often do not become obvious until deep into the adoption cycle, which means companies may already be too committed to change course by the time they come across challenges.

However, Mr Nakamura stated there is "light at the end of the tunnel" for businesses that have found themselves in this situation.

"The key lies in knowing your options, which usually involves leveraging the skillsets you already have in-house so that your big data strategy can continue up into the light," he said.

For instance, while understanding MapReduce is commonly seen as crucial to the success of Hadoop operations, this can be a challenge for companies that do not have personnel with these skills, or the resources to hire new experts.

However, they are ways around this, by using APIs or domain-specific languages that can hide MapReduce so developers do not have to fully master it in order to get the most out of Hadoop.

"For example, a modeling tool such as MicroStrategy, SAS, R or SPSS can use a DSL that will consume models and run them on Hadoop without needing to write Hive, Pig or MapReduce," Mr Nakamura continued.