Cloudera and Hortonworks — why the merger?

Last month, the merger of Cloudera and Hortonworks signposted a new era in the data market. But what exactly does two of the fiercest data rivals joining forces indicate?

Does the Hadoop market need to be a little less competitive to survive? Are Hadoop’s days numbered? Is there a bigger competitor to overcome?

Certainly, the big data market — and corresponding customer needs — have evolved since Hadoop became synonymous with Big Data. In the years since the two companies were founded, the landscape has changed significantly, and now includes a large number of companies with different strengths. But more significantly, Hadoop’s role has changed.

So, why have Cloudera and Hortonworks decided to merge?

Combine to survive?

Cloudera was co-founded in 2008 with skills and resources from Google, Facebook, and Yahoo. Its aim was to deliver a big data platform that was built on Hadoop. Three years after that, Hortonworks was created with essentially the same goal. So, two rivals were born.

However, it’s worth noting that in this time, neither company is yet to turn a profit. So the merger should allow the new combined firm to reduce costs and resources, and move towards a more cost-effective, profitable future.

“By merging Cloudera’s investments in data warehousing and machine learning with Hortonworks’ investments in end-to-end data management, we are generating a winning combination, which will establish the standard for hybrid cloud data management.”
Rob Bearden, president and CEO of Hortonworks

Is the cloud the true competition?

We can’t ignore the fact that the increased adoption of AWS, Azure, and Google Cloud services comes at the expense of on-premises infrastructure and software. In fact, some report that public cloud is quickly becoming the primary store for big data analytics for many industries.

Gartner, for example, believes that worldwide public cloud services will grow 18% to become a $247 billion business, and that cloud will account for the majority of analytics purchases by 2020.

Both Hortonworks and Cloudera recognized that their data management tools and data processing engines could work equally well with on-prem Hadoop clusters and cloud-based object stores like Amazon. Their merger could, therefore, spell a more productive era in this market for both companies.

Has Hadoop lost its hype?

Hadoop was perhaps one of the most over-hyped technologies of all time. Customers expected it to single-handedly answer all their Big Data woes. Now it’s becoming clear Hadoop is no silver bullet.

Thousands of companies spent time and resources putting their data into Hadoop. Yet, ever since the first MapReduce program more than a decade ago, the complexities of extracting value from this data in Hadoop have been widely acknowledged. Hadoop is a great storage repository, but ill-suited to fast, interactive applications.

As we know from Gartner’s Hype Cycle, first there is a period of exuberance that is either rational or irrational depending on various conditions. This was reflected in the silver-bullet phase of Hadoop.

This is generally followed by a corrective phase which is more or less severe depending on height the hype. This is finally followed by a period of fairly predictable growth. So, Hadoop is far from dead, it’s level of hype has simply plateaued.

Hadoop’s reduced hype spells good news for the big data market. Nowadays, more businesses are identifying how to use Hadoop for the right tasks, and figuring out its true capabilities. It just takes using best of breed tools for your specific needs rather than sticking with those that come with the standard distributions.

For the companies that have created vast Hadoop data lakes, they will continue to find ways to get true value from their stored data. It takes software like Kognitio, a YARN application that runs inside the Hadoop cluster, to allow business users to access the data efficiently in Hadoop. Once more companies recognize that Hadoop can answer its business users’ needs with the help of additional software, Hadoop’s true value will be realized.

A new opportunity for Hadoop

So where does this merger leave the Hadoop of the future?

Now, the Hadoop community has an opportunity to reset these overly-hyped expectations and help minimize its complexities, going forward. Now that Cloudera and Hortonworks developers can collaborate on their existing technologies, they may be better positioned to build a simplified, more flexible platform.

If the newly merged company, for example, converts Hadoop into a hybrid system, then there may be the potential for it to achieve some of its goals and claim more of the market. While the name may change, Hadoop as a concept will most likely remain; a set of open source programs and procedures that can be used as the backbone of Big Data operations.

Leave a Reply

Your email address will not be published nor used for any other purpose. Required fields are marked *