One of the biggest concerns for many businesses at the current time is the increasingly complex security landscape they have to deal with. There have been a large number of high-profile incidents recently that have raised awareness of the problems, with some commentators dubbing 2014 the 'year of the data breach'.

At the same time, one of the biggest developments in the technology world has been the emergence of big data analytics as an affordable and practical tool for more organisations. So with many companies having a greater ability to process information and receive insight into both their own operations and the wider market, could this help with firms' security issues?

One company that thinks so is Platfora, which has recently introduced a new Hadoop-based solution in partnership with MapR Technologies that aims to help companies identify security threats within their networks.

Speaking to Datanami, vice-president of products at Platfora Pete Schlampp explained the idea behind the project is to provide security analysts with the tools they need to detect threats in near real-time, without them needing the advanced skillsets of a data scientist or Hadoop administrator.

"What these companies are finding is that traditional security tools are architected in a very regimented way to deal with security data," he said, noting that they are often focused heavily on areas security professionals are familiar with, such as intrusion prevention systems and VPN logs. But in today's environment, this is far from the only data available.

"If part of the investigation involves other data sets – such as a bank that needs to look at transactions out of accounts – you can't mix those things together," Mr Schlampp explained. "One of the hallmarks of big data is variety, and being able to easily mix these data sets together to find these anomalies is pretty important."

The new solution, called Big Data Analytics for Security, aims to solve these issues by giving users a single platform from which they can monitor their security. The offering combines MapR's Hadoop distribution, a collection of machine learning algorithms from MapR, MapReduce-based analytic jobs, and Platfora's Hadoop application to collect, transform, and visualise large amounts of both structured and unstructured data.

But Platfora and MapR are not the only companies seeking to come up with big data solutions to the security headache. One startup, Niara, has received $20 million in investment funding to develop its own Hadoop-based cybersecurity offering.

Currently in beta testing and set for a full release later this year, it combines traditional heuristic techniques with newer machine learning technologies to enable companies to detect the traces that cybercriminals and malware invariably leave when they make their way across private networks.

Founder of the firm Sriram Ramachandran told Datanami that combining effective machine learning models with a parallel Hadoop cluster is essential as today's cybercriminals have become very good at disguising their activities, so they blend into the background of day-to-day activities.

What both Niara and Platfora/MapR are essentially doing is taking advantage of Hadoop's massive processing capabilities to sift through petabytes' worth of business data in order to spot extremely weak signs of a cyber attack.

The developments therefore illustrate the value big data analytics can bring to all parts of a business, as solutions such as Hadoop can provide capabilities far beyond those offered by traditional applications. In the years to come, businesses can expect Hadoop-based tools to play a bigger role across operations, including keeping networks and data safe.