As the AI revolution accelerates and the number of use cases for machine learning continues to grow, more and more machine learning teams are pursuing edge ML. Edge ML is the process of running machine learning models on devices– such as laptops, IoT devices, smart phones, cars, and more– as opposed to processing data on a cloud network.
Cloud networks offer a lot of benefits, but they aren’t suitable for every use case, and because edge ML enables models to make predictions as close as possible to the origin of the data, it comes with its own set of advantages.
The Evolution of Edge Computing
Over the past few years, the performance of machine learning models has increased tremendously with large-scale models demonstrating impressive predictive power. However, the large size of these models have limited their widespread adoption. Running them is costly, and they often have long training times, long interference times, and large memory usage requirements. However, recent advancements in model compression, such as pruning, quantization, and knowledge distillation, have allowed machine learning teams to represent models in an efficient format without any significant negative impact on their performance.
At the same time, edge devices have become increasingly powerful, and because these devices are closer to the data source, moving big volumes of data is no longer necessary. The result is that high-performance models can be run on edge devices at almost real-time speed.
By doing so, machine learning teams can decrease latency while increasing throughput. Previously, large model sizes and less powerful devices meant that ML teams had to use GPUs living on central servers to achieve high model throughput. The trade off was the cost and time of having to transfer the data back and forth. By running machine learning algorithms on edge devices, these teams can produce predictions much faster without having to transmit large amounts of data across a network.
Why Opt for Edge ML?
Multiple companies from nearly every industry– manufacturing, aerospace, energy, and more – are focusing on end-to-end processes around edge ML. For these companies, edge ML not only provides cost savings but can also give them a competitive advantage.
In use cases where the original source of the data comes from facilities in a remote location, such as manufacturing data or agricultural data, transferring data to a cloud is often challenging and costly because of constraints around access to connectivity. The length of time it takes to perform data transfers, especially when high-speed connectivity isn’t reliable, creates efficiencies in the machine learning pipeline.
However, even in less-remote locations where connectivity and speed are less of an issue, large data transfers often aren’t cost effective.
With distributed devices and edge ML, companies can optimize their processes by running live data into their models. They can scale model inference without having to buy larger servers. From a business perspective, it’s a huge benefit to be able to run inference close to the data source and in a decentralized fashion where the compute is paid by the end user instead of the company hosting the centralized server.
Edge ML, Active Learning, and Improved Model Performance
Once scaling inference is no longer an issue, companies can focus on collecting the correct data for the next training iteration. While collecting raw data isn’t difficult for most use cases, selecting which data to annotate for the next round of training becomes challenging when large volumes of data have been collected. The computer resources on edge devices can help identify which data points might be the most relevant for labeling. For example, if the use of a smartphone dismisses a prediction, that can be an indicator that the model predicted incorrectly, so annotating that particular piece of data and retraining the model based on a new label would help improve performance.
Automotus, which provides curb management solutions, likewise uses edge ML to iterate on its AI systems. The company is deploying a large-scale computer vision system on edge in a variety of cities throughout the U.S. Deploying models for street monitoring purposes is challenging because of the limitations created by the hardware present in the small cameras designed to monitor the streets. Having solved the deployment problem, however, Automotus has used edge ML to provide tremendous value to customers, such as municipalities and airports. Its AI software has helped to reduce parking turnover by 26 percent, double-parking hazards by 64 percent, and emissions from traffic congestion by 10 percent.
Because edge devices can help preselect the data that will be more valuable for the next model training iteration, edge ML is set to play an important role in end-to-end active learning pipelines. The most sophisticated companies are already using active learning as a strategy for closing AI’s prototype-production gap. Other companies are using active learning in a production environment to gain a competitive edge by way of constant model performance improvements. Multiple companies already run their production models on edge devices, cleverly monitoring their performance in the production environment and identifying potential model errors as a result. These errors are sent back for review or annotation so that the next training iteration has new, intelligently selected data samples to learn from.
With edge ML, the combination of intelligent curation and live data examples with running models close to the data source means that AI will continue to learn and improve rapidly even after it’s released in the wild. The result? Models will continue to adapt, learn, and predict accurately even as the data they encounter in the real world evolves or deviates from the data on which they were trained.
