For companies in many sensitive industries, managing their data in order to meet strict compliance rules can be a major headache. Often, regulators demand they keep large numbers of records on hand, which may be very hard to effectively comb through when needed.

It was noted by the Wall Street Journal that compliance applications vary widely, but often start with the company needing to identify what useful data it has available and then organising it into a useful format.

This is something that has proven difficult in the past, but the advent of big data analytics tools are changing this.

For instance, Lionel Wertz, director of corporate services at nuclear reactor company NuScale Power noted that until recently, his firm had operated a paper-based system, which keeps records for as long as 80 years beyond the lifetime of a reactor.

Now, however, it is moving to a data-driven system that will make it much easier to keep control of this information, which it is legally required to archive. This platform will enable it to  track and organise critical engineering information to be quickly searched and pared down.

“I’ll know how long I’ll keep my record for and when I can destroy it,” Mr Wertz said.

However, while some companies have found big data analytics is able to transform how they manage their compliance activities, many more are still struggling, largely because their systems are not equipped to deal with the large volumes of unstructured data that today’s tools generate.

Bob Rogers, chief data scientist for big data at Intel, told the Journal that while nearly every company is exploring how they can improve the use of data, only around half have the technology to pull in large amounts of unstructured data, and only a quarter can convert this into useful insight.

“Unstructured data by its very nature is an inference, it requires context as well as the data itself,” he said. “Determining what to measure and how to measure it has been a real struggle for compliance leaders.”

Among the most valuable information to these personnel will be consumer data and Internet of Things information, which can help improve their daily operations, as well as meeting compliance requirements.

For example, Bank of Tokyo-Mitsubishi UFJ turned to analytics to ensure it was meeting its obligations under Dodd Frank regulations for financial trading, which require all records and events surrounding a trade to be identified and collated.

However, the bank realised it could use this data to study its activities and look for common characteristics that indicate a good trade. It then used this insight to find out which sales practices were working and which weren’t, and reshaped its sales model accordingly.

In the past, this type of big data analysis would be out of reach of all but the largest companies, due to the amount of information to sift through and the computing power required to do this effectively. Nowadays, however, innovations such as cloud computing can give these capabilities to firms of any size, allowing compliance officers more opportunities than ever to not only improve their own activities, but assist the wider business.