Five critical data challenges organizations are facing in 2018

Modern business is largely data-driven, but this has opened up new challenges for forward-thinking organizations. Enterprises must come up with innovative ways to handle the volume, velocity and variety of data to ensure their insights are fast, accurate and synchronized.

What’s more, the need for skilled data workers is accelerating. Does that mean there’s an impending shortage of data analysts, or is supply meeting demand? Here’s the top 5 problems data-driven organizations face:

1. Learning big data platforms

Traditional data management platforms now buckle under the variety and volume of today’s datasets. To support it, there is a wide range of innovative data management tools and frameworks designed to support operational and analytical data processing.

For a novice, however, platforms like Hadoop can be incredibly complex. It takes time to understand all the components before implementing a system that’s production-ready. This remains a challenge well into the adoption process, and a team of skilled workers is often a prerequisite.

This rapid advancement of big data platforms has sparked a rising demand to hire the next generation of technical experts who can create and manage these ecosystems.

2. Data skills shortages

Whilst data is an incredibly valuable business asset, when it’s unstructured and chaotic it delivers minimal insight. Flexible, yet complex data platforms like Hadoop have been introduced to help store these vast data sets.

As such, we’re now facing a very real skills shortage; the European Commission projects we’ll need 346,000 more data scientists by 2020. Whilst this is worrying, it isn’t a new story; the shortage of data scientists has been well-documented over the years. But increasingly, businesses are facing a shortage of another key member of the data team – the data engineer. Data engineers are critical when a company intends to move its data science project into successful production.

The technical nature of managing distributed data stores — Hadoop, Amazon S3, and Azure Storage for example — has exponentially increased the demand for data engineers; they are simply the only one best-positioned to get true value from these systems.

3. Data security

Data breaches continue to be one of the most common problems experienced by organizations. In 2017, 83% suffered a data security incident, according to Clearswift.

What’s more, 2018 was the year of GDPR. So, when big data sets are collected, stored and queried, organizations must ensure they maintain the right balance between watertight security and utility of the data.

Yet, with an inherent lack of transparency in the Internet of Things (IoT) ecosystem, there are still a number of business devices potentially leaking sensitive data. With GDPR tightening the data processing protocols, businesses in 2018 must work at double-time to ensure they lock down the security of all their data-processing systems and devices.

Consider monitoring your database access activity and usage patterns in real time to detect data leakage, unauthorized SQL and big data transactions. Identify and classify your most sensitive business data and deploy strong authentication methods.

4. Management’s knowledge of data

With data driving more high-level business intelligence and strategic decisions, stakeholders now sit across a number of departments and functions. Driving the appropriate knowledge and ownership by management, however, can be frustrating for traditional businesses.

It appears that there’s still no single team or function considered responsible for managing data.

Research by Royal Mail found that, of businesses it surveyed:

  • 37% said data belonged to marketing
  • 37% said data belonged to central data management
  • 30% said data belonged to IT department

Data analysis is often made more complicated by the technology used to process it. Data distribution stores like Hadoop can be inflexible and difficult to use by a non-techie, so they can act as roadblocks to management utilizing data to meet their business challenges.

So, whilst organizations find their feet in business intelligence and data analysis, cooperation between management and different departments is increasingly vital. What’s more, using an SQL engine like Kognitio, that allows easy to use, industry standard data analysis and visualization tools to be connected to Hadoop, opens up the data to more business users, generating more buy in.

5. Deploying big data projects successfully

Big data may have been around for years, but collecting it alone is no marker of success; how many organizations can say they’ve successfully deployed a big data project, and generated value from their data?

WHISHWORKS conducted a survey among data strategists, architects and users. According to the survey, only 18% of companies in the UK have fully implemented a data project; many are still at the experimentation stage.

It confirms that Hadoop is the preferred open-source software to deal with big data, and is a good place to start experimenting with data. As data projects mature however, commercial platforms like Hortonworks, MapR Technologies and Cloudera are the preferred Hadoop-based solutions, with 75% of surveyed businesses employing or planning to deploy one of them for their data projects.

According to the survey, 70% of those who implemented a data project said their initiatives have been ‘somewhat successful’ so far. Only 30% state that their projects have been ‘very successful’.

In some cases, insights for multiple business users is incredibly slow directly on Hadoop-based platforms, potentially slowing the project’s success. Whilst Hadoop is a good starting point, therefore, a number of businesses may need additional software like Kognitio to unlock the speed and concurrency of their data insights on these platforms.

Leave a Reply

Your email address will not be published nor used for any other purpose. Required fields are marked *

SQL on Hadoop. Bring your BI to life.

Read how to transform Hadoop into your best BI platform

Download the guide