What will the future hold for Hadoop?

As one of the most hyped solutions for managing big data analytics, the future of Hadoop is something that is keenly discussed among professionals in this field. Since first being developed back in 2005, the technology has made huge strides to become one of the top solutions for handling large amounts of data.

However, the project is far from complete. In a recent interview with Computing magazine, co-creator of Hadoop Doug Cutting observed that it is still in the early stages of development, but swift progress is being made and growth is strong. He noted that Hadoop distributor Cloudera, where he is now chief architect, is seeing growth of around 100 per cent a year.

"We're growing about as fast as we can responsibly. You need to have a certain number of people with some mastery of it to expand sensibly," Mr Cutting said. "How long can we sustain that? I don't know. We're a long way from saturation but for most customers we're still a very small portion of their IT." 

One key task has been to make the technology more user-friendly, as a lack of available skills and experience with Hadoop has been identified as one of the main barriers to its development. Initially, the platform had a reputation of being very complex and difficult to understand, but with additions such as YARN, Impala and Spark, the technology has become more 'enterprise-ready'.

In the years to come, it is expected that organisations will find more use cases for Hadoop, and the technology will play a larger role in businesses' IT infrastructure as the ecosystem evolves.

Mr Cutting said: "There's a lot going on. I'm keen to see better support for transactions, systems where you are updating values in real-time and are still able to perform analytics on the same data."

He added that currently, many businesses may be so overwhelmed by the sheer number of big data options available to them, they neglect to consider how they will actually apply them to their operations. However, he said that as the number of proven use cases for tools like Hadoop grows, this will become less of an issue.

"It's in an early stage of maturity so that's not unexpected, but I think over time people are going to think about the functionality you've got in the distribution. You could have a SQL engine for analytics queries. You've got a NoSQL engine for reporting queries," Mr Cutting said.

He also observed that the technology is still young and there are improvements to be made in how it is packaged and delivered to users in order to make it a simple experience.

Mr Cutting said: "Training is a big part of our business, training operations and professional services, and we see those as enablers to the real long-term business, which is folks consuming the platform and deploying applications on it."

Once the novelty factor of Hadoop wears off and users get used to the idea that it will require more work upfront to determine what businesses want to do and what tools will be needed to do it, the flexibility offered by the technology will start to reap benefits.