A significant majority of federal US government agencies have adopted big data analytics tools in order to improve their cyber security capabilities, a new report has found.

The 'Navigating the Cybersecurity Equation' study, conducted by MeriTalk on behalf of Cloudera, revealed more than four-fifths of organisations (81 per cent) are using this technology in some form. Some 53 per cent of agencies reported big data has a role to play in their overall cyber security strategy, while 28 per cent stated they were using it in a more limited capacity.

However, many federal agencies are still struggling to keep up with the threats posed by hackers and other cyber criminals. Nearly six out of ten respondents (59 per cent) reported their organisation deals with a cyber security incident at least once a month due to their inability to effectively analyse the data they have available.

One of the most common problems is that agencies find it difficult to manage these resources. Some 88 per cent of respondents admitted they struggle to turn their data into useful cyber security intelligence, while almost half (49 per cent) said the sheer volume of information is overwhelming. 

Meanwhile, one in three agencies (33 per cent) said they don't have the right systems in place to gather the data they need, and 30 per cent found that by the time the information makes it to cyber security managers, it has become outdated.

Efforts to make the most of data are also hampered by budget issues, a lack of internal skills, privacy concerns and a lack of management support and awareness of such projects. As a result, more than 40 per cent of data gathered by federal agencies goes unanalysed.

Rocky DeStefano, cyber security subject matter expert at Cloudera, stated that as both internal and external cyber security threats are evolving on a daily basis, it is vital that government agencies are able to unlock the power of data in order to combat these dangers.

"Agencies need complete visibility into the data across their enterprise," he said. "These teams also need the ability to flexibly analyse that data in a meaningful timeframe so they can detect advanced threats quickly, identify the impact and reduce the associated risk."

Those agencies that are able to effectively adopt big data analytics tools have seen significant improvements in their cyber security defences. Some 84 per cent of big data users reported that their agency had successfully used the technology to prevent an attack, while 90 per cent have seen a decline in overall breaches.

These successes mean that agencies are keen to increase their investments in big data technology. Some 94 per cent of respondents stated their agency has plans to invest in big data analytics in the next two years, with key areas for improvement including technology infrastructure (61 per cent), hardware (52 per cent), and business intelligence tools/analytics (52 per cent).

Steve O'Keeffe, founder of MeriTalk, said: "Agencies face a perfect storm of cybersecurity threats. When you're headed into troubled waters, you need a weather forecast. Big data provides agencies with the visibility to ensure they don't end up the river without a paddle."