Many companies that are embarking on big data analytics projects will find themselves struggling to move beyond the pilot stage and roll out their initiatives on a wider scale, a new survey has found.

Research by Gartner identified this as one of the key challenges facing businesses when they undertake big data projects. It revealed that although nearly three-quarters of organisations are planning to invest in this technology or have already done so, just 15 per cent say they have deployed their big data projects at full production scale.

This is almost unchanged from when the same question was asked last year, when 14 per cent of firms stated they had achieved this.

Nick Heudecker, research director at Gartner, suggested one reason for this may be that big data initiatives are having to compete with other IT investments and are often treated as a lower priority.

Indeed, just 11 per cent of IT professionals at organisations that have invested in big data believed these initiatives were as important, or more important, than other projects, while nearly half (46 per cent) said they were less important.

"This could be due to the fact that many big data projects don't have a tangible return on investment that can be determined upfront," Mr Heudecker said. He added: "Another reason could be that the big data initiative is a part of a larger funded initiative. This will become more common as the term 'big data' fades away, and dealing with larger datasets and multiple data types continues to be the norm."

Another issue that can make it difficult for companies to move beyond the pilot stage is they do not have effective business leadership and involvement in these projects. What's more, many trial initiatives are developed using ad-hoc technologies and infrastructure that do not have the scalability or reliability needed for production-level deployment.

Overall, Gartner's survey revealed investments in big data continue to rise, with 48 per cent of organisations investing in the technology in 2016. This marked an increase of three percentage points from a year earlier.

However, the number of companies planning new investments dropped from 31 per cent in 2015 to 26 per cent in this year's survey.

Mr Heudecker said these signs of slowing growth may be an indicator that companies are rethinking how they look at big data analytics and integrate it into their operations.

"The big issue is not so much big data itself, but rather how it is used," he said. "While organisations have understood that big data is not just about a specific technology, they need to avoid thinking about big data as a separate effort."

One trend is that businesses are no longer thinking about the technology in vague terms, but are looking at specific problems and opportunities that it can address. As such, the success of such initiatives will depend on a "holistic strategy around business outcomes, skilled personnel, data and infrastructure", Mr Heudecker continued.