The inextricable link between IoT and machine learning

I met with a team of Microsoft AI researchers recently to discuss original adaptations of Resnet 50, a version of the convolutional network Microsoft used to win the Imagenet 2015 image recognition competition. The discussion about the scientists work caused me to reconsider the inextricable link between IoT and machine learning.

Control loops are a fundamental principal of the internet of things (IoT.) If then, then that (ITTT) has a long history in conditionally controlling things dating to the invention of the electric relay in the 1830s. Over time, single relays were combined into state machines, and later, relays became transistors. During the glamorous growth of computers in IT, consumer and mobile sectors, less glamorous ITTT computers have been applied to many use cases such as controlling machines in factories and performing lab experiments.

Makers, people who like to experiment and build things in their free time have bought over 12.5 million Raspberry Pis so that they could know if the postman delivered mail, how many times the cat went through the cat door, and if there is a water leak in the basement. IoT in simple terms is a ubiquitous ITTT system with intelligence. Ubiquity is like infinity, it is understood as a mathematical concept, but impossible to experience. En route to reaching the billions of IoT devices, forecasted, machine learning will fill the gaps in the resolution of IoT devices before IoT ubiquity is reached.

The simple case of a farmer watering his crops illustrates the role of machine learning. This is a fictional example based on my discussion with Microsoft’s scientists.

Water is an increasingly expensive resource. And, too little or too much and crop yields suffer. An old farmer might intuitively know which irrigation valves to turn on and when without a computer but farms are increasingly turning to technology to manage crop yields. Soil humidity sensors could be installed on the trillion acres of US farmland to create the ITTT control loop, but at what cost and what resolution? Farmers are turning to drones to map their fields with high-resolution images to understand crop health.

Image recognition is one of the most accurate subfields in machine learning. Train a machine learning model with images of cats and not cats and a trained model will recognize cats with accuracy in the high 90^th percentile. Train a model with images of fields with the correct soil humidity and fields with too much or too little, and it will predict with similar accuracy which parts of the fields need to be watered and could signal an IoT irrigation system to turn on or off. Installing, maintaining and wirelessly interconnecting soil humidity sensors on trillions of acres will not be necessary. Installing a fraction of the sensors in some of the fields would provide the ground truth drone images for training the model to recognize optimally watered crops.

Drones are the latest airborne technology used to inspect crops. A thread on the Drone deploy forum explains that DJI’s Phantom 3 drone can image about 100 acres on one battery. Use of the normalized difference vegetation index (NDVI) to interpret satellite images is referenced in the literature as early as 1972. NDVI is now used by farmers to interpret crop images taken with drones. But NDVI requires expert interpretation. Like so many medical imaging use cases such as diagnosing diabetic retinopathy that can be diagnosed with the same or better accuracy than an ophthalmologist with a machine learning model, the NDVI images could be diagnosed with a highly accurate model.

Part of the task of building these models has already been done with Resnet 50 available under open-source license. Pretrained vectors, trained on tens of millions of images is also available under open source license. The pretrained vectors provide an inherent understanding of images. In this use case, a high prediction accuracy of soil humidity would be achieved with the addition of tens of thousands of relevant crop images instead of millions, The layers of the convolutional network could be optimized to emphasize or de-emphasize certain features to increase accuracy.

Optimizing computational cost of the machine learning model like all other use cases there is a trade-off between accuracy and image resolution. The lower the resolution, the less computational cost. The range of resolution available from satellite images is 10m to drone images of 2cm. Also the lower the resolution that optimizes accuracy, the shorter the flight time of a drone to criss-cross a field and the longer the battery life.

In addition to saving the time and cost of deploying IoT devices and networks to interconnect them, machine learning could be a separate path to confirm an IoT system is working. A critical IoT device could fail and report a false condition. For instance, IoT sensors might fail to report critical conditions such as a fire, an unauthorized person entering or a door left open, but a machine learning model sampling a video feed could recognize the critical condition, all as adaptations of Resnet 50 or another convolution network.

Apple and Facebook have made a related point with the augmented reality smartphone cameras that can sense three-dimensional space. Instead of creating specialized depth sensing sensors, increasing the cost of a device and facing the three year lead time for consumers to adopt new phones, both companies built and trained machine learning models that detect three-dimensional space with ordinary cameras. The cost of imaging technology has dropped while resolution has increased. Machine learning hardware is getting faster and cheaper. Using existing image sources or creating new ones will be an often considered IoT alternative.