The number of businesses turning to Hadoop to assist with their big data analytics operations is continuing to grow. However, even the most sophisticated users are still struggling to get to grips with the operational complexity of the technology.

This is according to Wikibon, which noted that Hadoop has seen an "unprecedented" rate of innovation recently – primarily because it is an ecosystem rather than a single product, which enables providers to come up with their own solutions. This pace of development has encouraged a large number of businesses to investigate the capabilities of the platform.

According to the 2015 edition of Wikibon's Big Data Survey, 41 per cent of respondents reported they had at least one production deployment of Hadoop – a ten per cent increase from 18 months earlier.

However, analyst at the company George Gilbert observed that prospective users of the technology – as well as those already in the pilot or development stage – need to be aware that there are "no easy solutions" to getting the most out of Hadoop.

He noted the biggest challenge facing Hadoop users is making the technology manageable. As there are many moving parts involved in such a solution, this makes it a highly complex, difficult to control platform. If businesses do not appreciate this, they are likely to see their total cost of ownership (TCO) spiral as administrative overheads grow and users invest in solutions they later abandon.

One way to tackle these issues is to embrace cloud computing technology and turn to third-party experts to assist with the deployment. Mr Gilbert observed that Hadoop-as-a-Service solutions can simplify some of the management issues associated with the technology – although not all.

Turning to native cloud services like AWS, Azure and Google Cloud Platform can dramatically simplify management, as much of the burden of setting up and administrating a Hadoop system is placed on the service provider.

However, there is a trade-off for this. Mr Gilbert stated that in many cases, businesses that desire this level of simplicity will have to sacrifice choice and openness.

"All the cloud providers will provide ever more powerful DevOps tools to simplify development and operations of applications on their platforms," he said. "But as soon as customers want the ability to plug in specialised third-party functionality, that tooling will likely break down."

Therefore, enterprises may need to decide early whether they are prepared to accept a more limited level of functionality in exchange for reducing some of the complexity of Hadoop.

"Right now the customers with the skills to run Hadoop on-premises are internet-centric companies and traditional leading-edge enterprise IT customers such as banks, telcos, and large retailers," Mr Gilbert said. "Solving the TCO and manageability problem won’t be easy."