Human expertise ‘still required’ for successful big data deployments

Personnel with expertise in big data and programming will still be required to oversee and take control of advanced analytics solutions, despite the growing popularity of machine learning processes that are intended to be more automated than in the past.

This is according to new research from Evans Data Corporation, which revealed that the majority of developers working with machine learning solutions agree that direct intervention is still necessary on occasion.

Almost half of respondents (47 per cent) stated their machine learning tools require human input some of the time, while a slightly smaller number (44 per cent) stated such direct action is necessary most or all of the time. Just 2.6 per cent claimed they had successfully developed systems that are fully automated.

Director of research at Evans Data Corporation Michael Rasalan said many businesses view the potential for machine learning and artificial intelligence as an opportunity to develop computers that can process and act on data by themselves.

"In practice, development for machine learning is rather different from this fantasy," he added. "The overwhelming majority of developers involved in machine learning reveal that machine learning still requires a great deal of hands on involvement from programmers."

The survey also found that marketing departments are the number one users of big data in the enterprise, accounting for 14.4 per cent of total deployments. This was followed by IT teams (13.3 per cent), research and development (13 per cent) and sales (12.6 per cent). Overall, almost four out of ten big data analytics solutions (38.2 per cent) are being used in customer-facing departments.

It also revealed that businesses are dealing with a growing variety of data types and sources. The most common data sets developers are working with include sales and customer information, IT-based data analysis, informatics and financial transactions. However, system management, production and shop floor data, as well as web and social media-generated data, are also finding their place in businesses' data mix.

The result of this is that many companies are finding it a challenge to sift through this wealth of information to find the most useful data. The quality of information being gathered was named as a key problem area by almost a fifth of respondents (19.2 per cent), while 13.5 per cent had issues with the relevance of the information they collected and 12.67 per cent struggled to cope with the sheer volume.

Mr Rasalan observed: "A lot of the work in developing machine learning applications involves setting up big data systems and Hadoop implementations. But even after deployment, many developers will be creating and optimising algorithms required to analyse massive amounts of data. This process is continuous and requires direct human input and control most of the time."

Leave a Reply

Your email address will not be published nor used for any other purpose. Required fields are marked *