Are traditional big data dashboards on their way out?

Among the many topics up for discussions at last week's Strata + Hadoop World conference in New York, one area of debate was how firms should go about delivering their insights into the hands of business units in a form that is easy to understand for personnel without huge amounts of technical knowledge.

While demand for staff such as big data scientists to process information is higher than ever, many companies' drives to gather and analyse as much data as possible may mean they are left with a very complex system that is difficult for front-line employees to interpret.

As a result, new ways of processing and presenting this material are likely to be required in order for big data operations to be a success – and how this can be achieved was a subject touched on by several speakers at last week's conference.

Among those was chief executive and co-founder of ClearStory Data Sharmila Shahani-Mulligan, who said: "When all this information finally gets to the business, it is difficult for the business to understand what to glean out of the data. We know this has been a problem for several years now."

Therefore, she suggested that the traditional executive dashboard is likely to make way in the coming years, in favour of emerging techniques such as interactive storytelling that aim to give data context and meaning that is more readily apparent to viewers, PC World reports.

Also speaking at the event was Miriah Meyer, an assistant professor in the University of Utah's School of Computing, who gave a keynote address on the importance of effective visualisation in big data analytics solutions.

She said that for many organisations, the most challenging part of this process is gaining an understanding of a user's needs, and being able to translate this into a set of visualisation requirements. However, this is not a process that can be taken on lightly, as it is also one of the most important factors in a successful big data project.

"When done well, visualisation has the potential not only to support science but to also influence it," Prof Meyer said. “We have to move beyond thinking that visualisation is just about pretty pictures."

Ms Shahani-Mulligan also predicted that as part of this, dashboards are likely to become less useful, as many of the solutions currently in use have not updated to reflect the huge changes that have taken place in data analytics over the past decade. Common problems with these tools include the fact they are designed to look at data from perdetermined contexts and limit the amount of information that can be seen.

"You can't really dig in and see what is happening underneath the visuals," Ms Shahani-Mulligan said, noting that as the velocity of data increases and sources get updates faster, this is a problem that needs to be solved.

She estimated that companies using interactive storytelling could make the right decisions twice as quickly as those relying only on dashboards. 

ClearStory uses an Apache Spark-based solution as part of its interactive storytelling offering, and with this technology becoming more integrated with Hadoop, this could be something that many more businesses are able to take advantage of in the coming months and years.