New technologies ‘to be essential’ to big data

The growing capabilities of new technologies such as Hadoop are set to become more important than ever to companies looking to create a solid foundation on which to build data analytics solutions.

This is among the findings of a new survey from New Vantage Partners, which suggested Apache's software framework will be among the solutions firms are looking towards to help manage growing big data databases. Analytics tools such as SAS, Tableau and Revolution R were also highlighted by the company as gaining in adoption as organizations seek to support their big data efforts.

New Vantage's study also revealed that interest in big data analytics tools is on the rise. Its 2013 Big Data Executive Survey found 91 percent of senior staff say their firm has such an initiative in the planning or rollout stage. Six out of ten of these stated at least one solution has been fully implemented and a third have big data tools in operation across their business.

Randy Bean, co-founder and managing partner of New Vantage, said that since the last time the survey was conducted, in the fall of 2012, big data has "made the leap from hot topic to serious business adoption".

He added: "Big data differs from traditional approaches in the quantum leap in affordability, scale and variety of analytics it can support. Companies are radically expanding the scope, scale and impact of data and analytics in their day-to-day business processes and decision making."

This trend is only set to continue in the coming years, as 68 percent of executives report their organization will invest more than $1 million in big data solutions in 2013. By 2016, this figure will rise to 88 percent.

Even larger investments in big data will also become more common in the next few years, the survey found. This year, just 19 percent of firms will devote more than $10 million towards such initiatives, but this will rise to 50 percent by 2016. Large-scale investments of more than $50 million will also more than double, from six percent this year to 14 percent in 2016.

Big data trends will see firms looking more closely at all of the 'three Vs' of big data – variety, volume and velocity – which also feature much more prominently in business' thoughts. 

Last year, the need to integrate a wider variety of data sources was named by executives as their number one focus when it came to big data deployments. However, New Vantage observed: "The 2013 survey indicates a near equal distribution among integrating more data sources, analyzing larger volumes of data and analytical velocity – the speed by which organizations can obtain answers to critical business questions."

The key to success for many firms will rest on the speed of their solutions, with time-to-answer a top priority for many of the survey's respondents. Improving this enables businesses to make better, fact-based decisions and accelerate the speed at which they gain insight.

One answer to this that enterprises should investigate is therefore in-memory computing. This can greatly reduce the time it takes to answer queries, with complex operations reduced from hours to just minutes or seconds with the right technology.