Walmart turns to crowdsourcing to tackle big data skills shortage

One of the biggest challenges for many companies when they are embarking on a big data analytics project is ensuring they have people with the right skills on board to make the initiative a success.

Every year brings new warnings about the skills shortage. The situation is potentially getting worst with the growth of AI and big data analytics, the demand for skilled analysts has never been higher.

Enterprises need to be innovative in order to find the right personnel. Back in 2015, Smart Data Collective reported that retail giant, Walmart, was taking a different approach to the skills gap – employing crowdsourcing to assist with big data analytics.

Walmart turned to crowdsourced analytics competition platform Kaggle to help find top talent. This allows professional and ‘armchair’ data scientists to turn their skills to analytical problems submitted by companies, with the designer of the best solution being rewarded – in this case, with a job.

“The Kaggle competition created a buzz about Walmart and our analytics organisation,” explained Mandar Thakur, senior recruiter for the retailer’s technology division. “People always knew that Walmart generates and has a lot of data, but the best part was that this let people see how we are using it strategically.”

The contest allowed Walmart to recruit people based on their proven skills rather than their CV alone. For example, one successful entry would not have normally been considered for interview based on his CV. He had a strong background in physics but no formal analytics experience.

And it’s not only Walmart who has made use of Kaggle for recruitment. Telstra had a competition to predict service faults on Australia’s largest telecommunications network. AirBNB had a competition around new user bookings – where will a new guest book their first travel experience?

One thing’s for sure, the skills shortage is not going to be solved overnight. Upskilling existing employees, outsourcing to external specialist service providers, hiring students for internships focused on a specific analytics project are all potential options.

Another option for businesses is to use something like Kognitio. Our analytical platform reduces the requirement for highly skilled data scientists by simplifying the “productionization” of complex analytical queries. The data scientist creates the model in their language of choice and deploys it in parallel using Kognitio external scripts. Analytics and business users can then run the model themselves using their preferred analytics tools such as Tableau, without having to know anything about how the model is implemented. By creating and productionizing queries in this way, a data scientist can focus on new tasks and other users can continue to benefit from their productionized models. You can read more about Kognitio for Data Scientists here.

This post was originally published in August 2015 and has been updated for freshness, accuracy and comprehensiveness.

Leave a Reply

Your email address will not be published nor used for any other purpose. Required fields are marked *

How Kognitio works

Want to read detailed articles about how Kognitio’s analytical platform works? We’ve gathered all the content in one place.

Find out more