Optimizing TinyML applications 101: Part 2

This blog extends my TensorFlow Lite for Microcontrollers tutorial. I read through the TinyML book. The 16th chapter of the book details how and why to reduce energy usage in a TinyML project. I found it very useful and wanted to make a practical guide using it so that other people working on it can get started with energy usage optimization faster.

This blog in a nutshell

1. Why optimize energy usage?

2. Typical component power usages

3. Hardware choice

4. Measuring real power usage

5. Estimating power usage for a TinyML model

6. Improving Power Usage

7. Conclusion

Optimizing Energy usage

1. Why optimize energy usage?

The fact that embedded devices use a tiny amount of energy is their main benefit over desktop or mobile computers. A server CPU may need tens or even hundreds of watts, which necessitates the use of a cooling system and main power supply. Even a phone can use several watts and needs to be charged every day. Microcontrollers can operate at less than a milliwatt, which is more than a thousand times less power than a phone's CPU, and can therefore be powered for weeks, months, or even years by a simple coin battery.

The most difficult constraint you'll likely face while creating a TinyML product is probably battery life. Because it's frequently impractical to change or recharge batteries manually, your device's functional lifespan will be determined by how much energy it consumes and can store. The physical size of your product is often what restricts the battery capacity. This indicates that the primary factor you have control over to affect how long your equipment lasts is how much energy your system consumes.

2. Typical components power usages

The majority of desktop engineers have a general idea of how long different types of operations take, and they are aware that accessing a file from a solid-state drive will typically be faster than accessing a file from a spinning-disk drive and that a network request is likely to be slower than reading some data from RAM. It is considerably less frequent to have to consider how much energy various functionality requires, but you'll need to have some general guidelines for how much energy your activities use in order to develop a mental model and plan for power efficiency.

Here are some measurements observed for embedded components:

• A microphone sensor might use 300 microwatts (µW).
• Bluetooth Low Energy might use 40 mW.
• A 320 × 320-pixel monochrome image sensor (like the Himax HM01B0) could use 1 mW at 30 FPS.
• An Ambiq Cortex-M4F microcontroller might use 1 mW at 48 MHz clock rate.
• An accelerometer might use 1 mW.

These figures will vary somewhat depending on the precise parts you employ, but it's helpful to keep them in mind so that you at least understand the general ratios of various operations. In a nutshell, radio consumes a lot more power than other features that might be required in an embedded product. Additionally, it appears that the energy requirements for sensors and processors are decreasing far more quickly than those for communications, so it's conceivable that the gap may widen much further in the future. You need to consider how much energy you can store or harvest to power them after you have an estimate of what the active components in your system are going to need.

Here are some rough figures for power sources:

• A CR2032 coin battery might hold 2, 500 J. This means that if your system is using one mW of power on average, you could hope to get roughly a month of use.
• An AA battery might have 15, 000 J, giving a six-month lifetime for a 1 mW system.
• Harvesting temperature differences from an industrial machine could yield 1 to 10 mW per square centimeter.
• Power from indoor light could give 10 µW per square centimeter.
• Outdoor light might enable you to harvest 10 mW for each square centimeter.

3. Hardware Choice

When you have a rough idea of what kinds of components you might use in your product, you’ll need to look at real parts you can purchase.

It's a good idea to start by reading websites like SparkFun, Arduino, or AdaFruit if you're seeking for something that is well documented and available to hobbyists.

• Advantage: These websites provide accessories with tutorials, drivers, and guidance on connecting to other accessories. They are also the perfect place to begin prototyping because you might be able to obtain a full system that is already populated with all of the necessary information.
• Disadvantage: You will have a more limited selection, the integrated systems might not be optimized for overall power usage, and you will be paying a premium for the extra resources.

You can try electronics providers like Digi-Key, Mouser Electronics, or even Alibaba for more options and lower prices but without valuable support. The fact that datasheets for each of their items are provided is something that all of these websites have in common. These provide a variety of information on each component, including mechanical information on the size of the chip and its pins as well as information on how to deliver clock signals.

4. Measuring Real Power usage

You need to put together a complete system from the collection of components once you acquire them. The ability to test the actual power consumption is one advantage of having a complete system. Planning can be assisted by datasheets and estimations, but there are always variables that defy simple modeling, and integration testing often reveals substantially higher power usage than anticipated.

There are several tools available for measuring a system's power consumption, and being able to use a multimeter can be quite beneficial. However, the most trustworthy technique is to insert a battery with a known capacity into the apparatus and then observe how long it lasts. After all, this is what you really care about, and while you may be aiming for a lifetime of months or years, it's likely that your initial efforts will last only a few hours or days.

The benefit of using an experimental technique is that it captures all the factors that matter, such as failures that occur when the voltage drops too low, which are presumably not captured by simple modelling calculations. Additionally, it's so straightforward that even a software engineer could do it!

5. Estimating Power usage for a TinyML model

The simplest technique to determine how much energy a model will consume on a specific device is to measure the latency for making one inference, multiply that value by the system's typical power consumption during that time period, and finally calculate the energy consumption.

You probably won't have precise numbers for latency and power consumption at the beginning of a project, but you can estimate them. You can roughly predict how long it will take to run a model if you know how many arithmetic operations it needs and how many operations a processor can handle per second.

Datasheets will typically provide values for a device's power consumption at a specific frequency and voltage, but be aware that they most likely won't cover common components of the entire system, such as memory or peripherals.

6. Improving Power Usage

Knowing the approximate lifespan of your system gets you probably considering how to make it better. Hardware tweaks like turning off modules you don't require or replacing components might be able to help, but those require an electrical engineering background. Fortunately, there are a few common practices that don't require an electrical engineering background but are still quite beneficial. These methods imply that the microcontroller itself is using the majority of the power because they are software-focused. You must do a hardware investigation if your device's sensors or other components are power hogs.

6.a Duty Cycling

Almost all embedded processors can go into a sleep state where they don't perform any work and consume very little power, but they can still wake up either after a period of time or when an outside signal enters the system.

The CPU will spend more time in a low-power mode if sleep cycles are added in between inference calls, which is one of the simplest ways to reduce power. In the embedded world, this is frequently referred to as duty cycling. You might be concerned that this prevents continuous sensor data collection, however, a lot of contemporary microcontrollers feature direct memory access capabilities that allow them to continually sample analog-to-digital converters and store the results in memory without involving the main CPU.

Similarly, you might be able to drastically reduce the processor's power consumption by lowering the frequency at which it executes instructions. This will make the processor run more slowly.

Duty cycling and frequency reduction allow for the trade-off of computation for power consumption. In real life, this means that you can trade off a higher power budget for a lower latency of your software if you can. If you wish to use less power, even if you can finish in the allocated time, search for techniques to optimise latency.

6.b Cascading Design

The fact that machine learning makes it simple to scale up or down the amount of computation and storage resources needed, and that the accuracy will typically deteriorate smoothly, is one of its major advantages over traditional procedural programming. With manually designed methods, this is more challenging to accomplish because there are frequently hidden factors that may be changed to affect these qualities.

This implies that you can build a cascade of models, as the names imply. Even though sensor data is not very accurate, it can be tweaked to have a high possibility of triggering when a specific condition is present by feeding it into a very small model with minimum computational requirements. If the outcome suggests that a significant event has just occurred, the same inputs can be used to feed a more complicated model to create a more precise outcome. This procedure might be repeated for a number of additional steps.

This is helpful since the inaccurate but small model can fit inside an embedded system that uses very little power, and running it continuously won't use up much power. A more potent mechanism can be activated in response to the detection of a possible event, and so on down the cascade. The more powerful systems are only used for a tiny portion of the time, therefore their power consumption is within budget. This implies that an embedded product can still be able to accomplish its objectives even if it is unable to host a model that is precise enough to be useful on its own.

7. Conclusion

I thank my GSoC mentor, Paul Ruiz, for guiding me throughout the project!