Addressing the Great IoT Analytics Skills Gap

Expect to see a huge uptick in demand for people who have technology and business skills related to the Internet of Things (IoT), as organizations continue to ramp up their IoT projects in a big way.

A new report by 451 Research notes that finding IoT-skilled workers is a big challenge. Nearly half of the 575 IT and IoT decision makers the firm surveyed, primarily in North America and Europe, said they face a skills shortage for IoT-related tasks.

The skills companies need to acquire include expertise in areas such as cyber security, networking, device hardware, applications, and overall management of IoT strategy. But perhaps nowhere will demand be greater in the coming years than in areas related to data analytics. As companies seek to use IoT data to predict outcomes,  prevent failures, optimize operations and develop new products, advanced analytics competency—including artificial intelligence (AI) and machine learning (ML)—will be critical to their success.

In addition to data scientists, organizations will need people who can assist with data collection, hygiene, cleansing and transformation of IoT data sets into a form in which they can be processed by analytics algorithms, providing timely operational insights.

Skills gap

Individuals with these skills are hard to come by, especially people who combine traditional data analytics backgrounds with competence in areas such as artificial intelligence, machine learning and IoT streaming data processing—all of which will be integral to developing and implementing a successful IoT strategy.

Unlike traditional business intelligence (BI), where companies run a periodic batch process and business users produce and consume reports, many IoT use-cases demand near real-time operational insight. In the case of IoT AI/ML algorithms for operational information (whether in factory automation, smart energy, fleet management or healthcare), the algorithms are continuously consuming streaming data and generating insight. And that insight must be used in a timely manner to optimize operational processes.

As enterprises delve more deeply into IoT, there will be a growing need for an operational intelligence-oriented data analyst who is also an “AI/ML data engineer.” Although not primarily responsible for algorithm development or selection, this new persona must be constantly involved in the operational fine tuning and evaluating data sets to ensure that the highest quality insight is produced. This persona needs to understand database schemas, transformation/cleansing approaches, big data and ML pipelines, streaming data processing, data visualization and IoT security. The persona also must be able to speak a common language with both business users and data scientists, who do the heavy lifting on the mathematics of the algorithm, to facilitate effective collaboration.

Unfortunately, at this point, almost no one has the ability or experience to take on such a role. The reason is quite simple. This area is new, uncharted territory for both educational institutions and enterprises alike.

Don’t look to data scientists

Colleges and universities will soon need to start approaching this IoT analytics skill set as an entirely new discipline, in order to meet what will almost certainly be a rapidly growing market demand. But in the meantime, this skills gap presents a significant challenge for companies, and it could even jeopardize their ability to fully realize the promise of IoT.

One possible solution is to train existing data analysts, ETL developers and other data-oriented professionals in these newer technologies. But that might also prove to be challenging.

Don’t expect to turn to data scientists to fill the need. In fact, the mathematically-oriented data scientist skill set is the principal constraining factor in the use of AI/ML for creation of IoT insights today. We simply do not have, nor will we be able to create, enough talent with this skill to keep up with the demand.

Since IoT data is a subset of the broader category referred to as “big data,” a quick look at the daily experience of big data scientists serves to further highlight the IoT dilemma. According to a recent survey by Forbes.com, “data preparation accounts for 80% of the work of data scientists” while “76% of data scientists view data preparation as the least enjoyable part of their work.”

This explains why there is a need to separate the mathematically-oriented, algorithm development skill set, usually found in a Ph.D. mathematician or data scientist, from the operationally-oriented skill set of a new persona: an AI/ML-literate data analyst/engineer. If properly trained, this individual can do 80 percent of the work now performed by data scientists. The result will be that organizations will have more leverage over the highly constrained data scientist resources and a better ability scale their use of AI and ML for IoT use-cases.

Next-generation IoT analytics tools

This new IoT analytics persona will need to rely on a generation of self-service analytics tools that will do the heavy lifting when it comes to collecting, cleansing, organizing and analyzing data generated by IoT systems. Unlike the popular tools and programming languages that are designed for (and preferred by) data scientists, these new tools must be easily learned by skilled IoT specialists who are neither programmers nor mathematicians.

The new AI/ML analyst/engineer should be able to assist in data preparation (under the guidance of a data scientist); select appropriate models and to compatible data sets and problems; build the feeding pipeline needed to operationalize model in each new scenario; do modest evaluation of model performance and tune basic parameters; and escalate to a data scientist if the model is not performing as expected. Such a tool would facilitate seamless collaboration between IoT data analyst/engineers and data scientists, allowing each to contribute to the rapid creation of advanced operational insights that enterprises need in order to get the most out of their IoT initiatives.

Finding a solution to the analytics skills gap is not a trivial consideration for organizations. Without the ability to analyze the enormous amounts of data to be generated by their connected devices and efficiently operationalize the insights generated, enterprises will struggle to fully realize the true value of IoT.