speed, in-memory processing

One of the key factors that will separate the best-performing big data programmes from those that fail will be how quickly they can deliver the results businesses need.

 

This should be at the heart of any such initiative – after all, velocity is one of the core ‘three Vs’ of big data, along with volume and variety. But what will this mean in practical terms for a company, and how can they go about ensuring they are in the right position to respond to these demands?

 

The need to acquire, analyse and interpret data quickly is only set to grow as the volume of information increases and business demands change. For instance, developments such as the Internet of Things and social media can provide organisations with huge amounts of data, but if businesses are not able to gain insights from these sources in a timely manner, efforts to gather it will be wasted – the value of data has a best-before sticker!

 

What does ‘fast data’ mean to you?

 

It is important to recognise that each business will have different requirements for data speed, and what will be timely for one enterprise may be painfully slow for another. In general, when businesses talk about speed, what they mean is the need for a quick response to any query they may have – similar to what they expect when they ask a question of Google.

 

But what this translates to may depend on what a company intends to do with their data. For example, an online retailer may view real-time as within minutes, which could allow it to dynamically alter pricing based on available stocks and demand, whereas a bank may need answers in under a second to spot potentially high risk or fraudulent transactions.

 

This could have an impact on how organisations approach big data. Will they be required to provide an answer to a single query in a matter of milliseconds? Or will their requirements be more focused on boosting their overall throughput by having many actions occurring within the same timeframe, to deal with complex demands?

 

Meeting the challenges

 

Whatever the demands, a timely response to data queries and analysis is essential, and there are a few key challenges that need to be addressed to achieve this. For instance, one requirement will be to ensure low latency and high throughput for any activities.

 

This is essential in guaranteeing business efficiency, but in the past, this was a challenge that could only be met if enterprises were prepared to devote huge amounts of computing resources to the task. This was very expensive and therefore fast results were the preserve of only the largest companies.

 

However, advances made in the last few years in both software and hardware have made fast speeds much more affordable, bringing them within reach of a much wider range of businesses. While it may still not necessarily be cheap to get instant results, it is a far more achievable goal than in previous years.

 

The resources you need

 

Factors such as falling prices for servers and components like memory have played a key role in this, enabling organisations to acquire the hardware they need to support this. Scalable technologies such as in-memory computing therefore represent real possibilities for companies to increase the speed at which they gain insights from their data by changing how servers transfer, store and process data.

 

At the same time, a range of open-source software tools such as Hadoop will also play a vital role in storing and interpreting large quantities of information, but Hadoop is still batch oriented and other technology is required to support low latency, high throughput analytics. Gaining access, taping into, these resources will be vital if companies are to make sure their big data analytics ambitions deliver effectively.

 

Of course, simply having the right hardware and software tools to process data and deliver results quickly is only half the task. In order to make the most of this, businesses will still need to invest in understanding what questions to ask and how to translate the insights they get into real-world actions.