Taming the ‘Wild West’ of business intelligence

One of the key challenges for many firms in the current environment will be attempting to get the huge and ever-increasing amounts of business intelligence (BI) data they have to deal with under control.

Much of the data enterprises now have access to – such as social media updates, online reviews and videos – are unstructured and so may often go ignored by businesses trying to set up data analytics solutions, as they do not know how to extract useful results from them.

However, being able to harness this information will be a crucial first step in the transformation of companies away from traditional data-gathering architectures and towards the 'brave new world' of big data analytics.

Central to this will be tools such as Apache Hadoop, which was described by Paul Groom, chief innovation officer at Kognitio, as now being a key component of the next generation of data solutions. He said this has matured beyond just being the "latest toy" to be played with by IT professionals, adding: "Business users are now seeking to implement Hadoop as a part of their infrastructure for new insights and new applications that weren't possible or practical before."

But this brings its own problems, which will be something Mr Groom is focusing on in his talk at the Hadoop Summit 2013, which is taking place later this week in San Jose, California. He likened the current state of Hadoop adoption to the 'Wild West', as many businesses scramble to get on board with the technology, despite not having a clear idea of how it will work for them.

"We see a wide variety of localized approaches to implementation, with business users leading the way out of need and trying to create their own destiny – a new frontier," he said.

The key to making a success of this will therefore be to ensure the change takes place rapidly and businesses are fully aware of the tools that will be needed to ensure this goes smoothly. Mr Groom noted that smaller organizations in particular often have difficulty adapting to the three Vs of big data – volume, velocity and variety. 

Hadoop is an attractive solution to these firms as it offers the opportunity to capture the data easily and let companies worry about sorting it out later, while at the same time reducing costs for organizations.

However, more work needs to be done to ensure Hadoop integrates effectively. Mr Groom said it is up to BI vendors to improve and deepen their integration with the Hadoop platform so that it works well within the BI ecosystem. "In principle, they connect, but not with the attributes and capabilities that business users expect, such as rapid access and freedom of access to drill on-demand."

He added that the world of BI is moving away from SQL-type queries towards analytics and a higher priority is being placed on forecasting than in the past. Hadoop plays a key role in this, with Mr Groom comparing it to the railroads that brought a sense of order to the Wild West.

"In the same fashion, you can think of Hadoop as establishing a similar sense of forward-thinking order, which is democratizing BI and making it available to far more people and firms," he said.