Hadoop ‘no longer optional’, Forrester says

Businesses that have avoided exploring the potential of Hadoop for improving their big data analytics solutions will find themselves with no excuses for not adopting the technology in the coming year, as it is set to become mandatory for any firm that wishes to be successful.

This is according to a new report by Forrester Research, which offered its key predictions for the state of the technology in 2015. Primary among these, it said the new era of "Hadooponomics" means deploying the tools is no longer optional.

"The jury is in. Hadoop has been found not guilty of being an over-hyped open source platform," Forrester said. "Hadoop has proven real enterprise value in any number of use cases including data lakes, traditional and advanced analytics, ETL-less ETL, active-archive, and even some transactional applications."

Report author Mike Gualtieri said all these use cases will be driven by Hadoop's ability to linearly scale both data storage and data processing, which will mean any chief information officers that still find themselves "dazed and confused" by Hadoop need to make the technology a priority for the year ahead.

Among Forrester's other predictions for Hadoop, it forecast that the challenges posed by the current skills shortage are set to ease in 2015 as knowledge of the platform grows.

Mr Gualtieri observed that despite the hype, Hadoop is not that hard to understand, as at its heart it is just a file system and computing platform, with APIs that rely on familiar Java technology.

"The shortage of Hadoop skills will quickly disappear as enterprises turn to their existing application development teams to implement projects such as filling data lakes and developing MapReduce jobs using Java," he stated.

Meanwhile, those personnel who are already familiar with business intelligence will find that SQL on Hadoop opens news doors to do more with their data, while retaining the access that are comfortable with.    

Indeed, the report forecasts that SQL will become Hadoop's killer app, as fast, ANSI-compliant solutions will create opportunities for Hadoop to become a useful data platform for enterprises, due to professionals' existing familiarity with the technology. Forrester said this will provide a sandbox for analysis of data that is not currently accessible.

The closing of the skills gap will be very good news for businesses, as they will no longer have to compete for high-priced Hadoop consultants in order to get projects completed, while the additional skills needed for complicated applications such as predictive analytics can be built up over time.

Mr Gualtieri said: "Hadoop projects will get done faster because the enterprise’s very own application developers and operations professionals know the data, the integration points, the applications, and the business challenges." 

However, Forrester also noted that Hadoop's emergence will not only benefit analytics operations. Factors such as Hadoop's YARN management layer, the HBase transactional NoSQL database, and the Apache Spark in-memory platform will turn Hadoop into an application platform that can run code in combination with data. 

This will see the technology used as a single solution for a range of applications, meaning vendors offering services such as middleware, databases, search engines, integration tools, and other software infrastructure will need to develop versions that can run natively inside Hadoop.