Hadoop to be data center mainstay ‘for next decade’

Big data analytics tool Hadoop is likely to remain a market-leading solution for at least the next ten years, its creator has stated.

Doug Cutting told the SD Times in an interview ahead of last week's Big Data Technology Conference in San Francisco that it was be a "mainstay platform" for data centers for the foreseeable future.

He said: "Ten years from now, we're going to continue to see Hadoop gaining market share as the primary player. All the trends look that way. There's nothing that undermines it."

One of the key reasons for this is its flexibility and versatility, Mr Cutting continued. The Hadoop creator noted that even if developers have a need that the standard platform doesn't fit, it is easy to implement the changes needed, so business will not have to abandon Hadoop in the coming years due to their evolving requirements, as it will be able to change with them.

Mr Cutting also offered some detail on the new and upgraded features included in Hadoop 2, which received its general availability release last week. He said one of the main priorities is to fill in some of the gaps that were present in the first version of the platform, now that its capabilities are proven and businesses understand the potential of the technology.

Hadoop's strategy was always to offer scalability and utility, and not worry about having all the features from day one, Mr Cutting said. He added: "I think people often get distracted by not realizing what is a critical thing. Triage is the term they use in medicine. A lot of projects don't have enough triage." 

Now that these critical goals have been achieved, Hadoop can focus on other areas to add value to the platform. For instance, the developers have been doing a lot of work to boost security, while Hadoop 2 also adds the ability to run batch jobs outside of MapReduce. This was something the team had talked about from fairly early on, Mr Cutting told SD Times, but it was more important to get MapReduce stable and working well first.

In the coming years, the addition of search to the platform is something Mr Cutting said he is "really excited about." He predicted that in the near future, people will wonder how they ever conducted searches without Hadoop.

"Predicting what the next huge thing will be is hard to say. A lot of what we spend time on these days is polish: filling in features enterprises need so they can handle their compliance issues," he said. "These are things that are somewhat boring, but important in supporting auditability."

However, looking forward to Hadoop 3, he suggested that improved multi-tenancy support will be a focus, as this is something users will continue to demand. The new YARN capabilities in version 2 of the platform is a "huge step forward" in this, but there is much work still to be done, Mr Cutting said.

Hadoop is also likely to support online transaction processing in the future, though this is still a way off and might be related to features such as multi-data center support.