Home Business framework Tips and tricks for deploying TinyML

Tips and tricks for deploying TinyML


TinyML is a generic approach to reduce AI models and applications to run on smaller devices, including microcontrollers, cheap processors, and low-cost AI chipsets.

While most AI development tools focus on building larger, better performing models, deploying TinyML models forces developers to think about doing more with less. TinyML apps are often designed to run on battery-limited devices with a few milliwatts of power, a few hundred kilobytes of RAM, and slower clock cycles. Teams need to do more planning up front to meet these stringent requirements. Developers of TinyML applications should consider hardware, software, and data management and how these will fit together during prototyping and scaling.

ABI Research predicts that the number of TinyML devices will grow from 15.2 million shipments in 2020 to a total of 2.5 billion by 2030. This promises many opportunities for developers who have learned to deploy TinyML applications. .

Sang Won Lee, CEO of Embedded AI Platform Qeexo, said, “Most of the work is similar to building a typical ML model, but TinyML has two additional steps: converting the model to C code and compile for the target hardware. This is because TinyML deployments are geared towards small microcontrollers, which are not designed to run heavy Python code.

It is also essential to plan how TinyML applications can deliver varying results in different environments. Lee said TinyML apps typically work with data from sensors that are heavily dependent on the surrounding environment. As the environment changes, the sensor data also changes. Therefore, teams should plan to re-optimize models in different environments.

What does getting started with TinyML involve?

AI developers may want to familiarize themselves with C / C ++ and embedded systems programming to understand the basics of deploying TinyML software on constrained hardware.

“Some familiarity with the general principles of machine learning, programming in embedded systems, microcontrollers and working with hardware microcontroller boards is required,” said Qasim Iqbal, chief software architect at the developer of sub. – Autonomous sailors Terradepth.

Good products to support TinyML deployments include the Arduino Nano 33 BLE Sense, SparkFun Edge, and the STMicroelectronics STM32 Discovery Kit. Second, a laptop or desktop computer with a USB port is required for interfacing. Third, it’s fun to experiment by equipping the hardware with a microphone, accelerometer, or camera. Finally, Keras and Jupyter Notebooks software packages may be required to train a model on a separate computer before that model is moved to a microcontroller for execution and inference.

Iqbal also recommends learning preprocessing tools that transform raw input data to feed it to a TensorFlow Lite interpreter. Then, a post-processing module can modify the inferences of the model, interpret them and make decisions. Once this operation is complete, an output management step can be implemented to respond to the predictions using the hardware and software capabilities of the device.

Before getting too serious, a few demo projects can help developers understand the implications of various TinyML constraints. In addition to limitations on RAM and clock speed, developers may also want to explore the limitations of lightweight Linux distributions that run on their target platforms. These often have limited operating system and system library support that they would expect on larger Linux systems.

“Sound decisions about the right device hardware, software support, machine learning model architecture, and general software considerations are important,” Iqbal said.

It’s useful to determine whether a microcontroller will support the intended application or whether larger devices, such as Nvidia’s Jetson series of devices, might perform better.

Combine hardware and software

Developers new to TinyML software may consider investigating the community behind each TinyML tool before becoming too attached to a particular tool.

“Very often you won’t be able to find answers to your questions in the official documentation,” said Jakub Lukaszewicz, head of AI for construction technology platform AI Clearing. Lukaszewicz often resorted to surfing the Internet, Stack Overflow, or specialized forums to find answers. If the ecosystem around the platform is large and active enough, it’s easier to find people with similar issues and learn how to solve them.

It is also useful to study the material available before diving too deep.

“The sad news is that in post-pandemic reality, delivery times can be long and you may end up with a limited choice of what’s currently available on the shelves,” Lukaszewicz said.

Building a large-scale TinyML application begins with writing a detailed description of the application and its requirements.

After getting the table, the next step is to choose the ML framework to work with. Lukaszewicz said that TensorFlow Lite is the most popular framework right now, but PyTorch Mobile is gaining traction. Finally, you want to find some tutorials or dummy projects using the ML framework and the board of your choice to see how the parts fit together.

Beware of changes in frameworks and hardware which can create issues. Lukaszewicz often struggled with outdated documentation and things not working as they should.

“Often times the platform has been tested against a given version of a framework, such as TensorFlow Lite, but struggled with the most recent one,” he said.

In such cases, it recommends upgrading to the latest supported version of the framework and rerunning your model.

Another issue is unsupported operations or insufficient memory to fit the model. Ideally, developers should take a standard model and run it on a microcontroller without much hassle. “Unfortunately, this is often not the case with TinyML,” said Lukaszewicz.

He recommends first trying out models that have been proven on the board of your choice. He often found that an advanced model uses math operations that are not yet supported on some devices. In such a scenario, you would have to change the architecture of the network, replace those operations with supported operations, and recycle the model, hoping that all of this would not sacrifice its quality. Reading forums and tutorials is a great way to see what works and what doesn’t on any given platform.

Deploy AI at the edge

Developers should consider all viable approaches when deploying TinyML as a robust and scalable application rather than a proof of concept. Building a large-scale TinyML application begins with writing a detailed description of the application and its requirements. This will help guide the selection of sensor hardware and software.

“Typically, a business starts with a use case that pushes it towards TinyML and from there begins to identify a solution that meets its needs,” said Jason Shepherd, vice president of ecosystem. at Zededa and a member of the Linux Foundation Edge board.

Given the limited nature of the peripherals involved, there is an extremely tight coupling between the software and the capabilities of the underlying hardware. This requires in-depth knowledge of on-board software development for computation optimization and power management. Shepherd said organizations often build TinyML apps directly instead of purchasing infrastructure, especially early on.

It’s a great way to learn how all the pieces fit together, but many teams find it’s more complicated than they thought, especially when working out the details of the AI ​​deployment at scale, support its entire lifecycle and integrate it into additional use cases in the future. It is worth exploring new tools from vendors such as Latent AI and Edge Impulse to simplify the development of AI models optimized for the silicon on which they are deployed.

Organizations that decide to build these applications in-house need a mix of integrated hardware and software developers who understand the tradeoffs inherent in using highly constrained hardware. Shepherd said key specialties should include the following:

  • understand the training and optimization of models;
  • develop efficient software architecture and code;
  • optimization of energy management;
  • coping with coerced radio;
  • networking technologies; and
  • implement security without the resources available on more efficient hardware.

Organizations should consider the privacy and security implications of deploying TinyML applications in the field to be successful in the long term. While TinyML apps hold promise, they could also open up new problems – and bottlenecks if businesses aren’t careful.

“The success of advanced artificial intelligence as a whole and of TinyML will require our concerted collaboration and alignment to move the industry forward while protecting us from potential abuse along the way,” said Shepherd.