Hadoop growth highlights skills shortage

As Hadoop becomes more mature and a growing number of organizations look to adopt the technology, this is highlighting the need for more personnel with the skills to make the most of the tool.

Interest in the big data analytics solution has been particularly high since the launch of Hadoop 2 in October, which introduced a range of new features such as YARN. Apache describes this redesigned resource manager as a large-scale, distributed operating system that allows multiple big data applications to run simultaneously.

Tech Target reports, however, that these new features will require a high level of understanding if companies are to get the full benefit from Hadoop, which is something many organizations are currently lacking.

Dr Eric Little, vice-president and chief scientist at Modus Operandi, a Melbourne, Florida, company providing data management and analysis technology to government agencies, told the publication that "there's a huge skills shortage" in the market at the moment.

"Even the large businesses can't find these people," he continued. "What that means is that you're competing against IBM and Apple and Google and Amazon and Yahoo! for the exact same pool of resources."

However, there are ways that firms can address this skills gap and improve their data analytics performance. While recruiting Hadoop specialists is one way to get quick results, even if companies can find the right personnel, this is likely to be an expensive option.

Another choice is to nurture skills inside the company. This may take a little longer, but it also has the advantage that personnel will already understand the unique needs of the organization and will be able to see how Hadoop can fit into this.

Tech Target noted that many businesses have already started looking within their own ranks for people to help their Hadoop implementations – and these employees do not necessarily come from the IT department, as there are a growing number of business analysts experimenting with big data technologies and developing their own skill sets.

Gartner analyst Merv Adrian told the publication: "They are the unnoticed coders, if you will. They have been building stuff on their own. They've done it on Amazon, or they've downloaded free distributions and played with them." 

Such individuals may have a headstart on others, so being able to spot available skills when recruiting from within – and not being constrained by traditional ideas of where to find this talent – will be increasingly important for firms.

Mr Little said individuals with a strong math background can be a good place to start. "My experiences are that the people who are often quite good at this have to be real algorithm developers, which means they have to be pretty strong applied mathematicians."

Equipping these personnel with the tools found in Hadoop 2 can be a great way for businesses to improve their analytics capabilities, though they need to be aware of what they can realistically do with it.

George Corugedo, co-founder and chief technology officer at RedPoint Global, told the publication: "With Hadoop, as with any new technology, it is important to build credibility and skill and not fall prey to the hype and over reach."