EON Tuner: AutoML in Embedded ML with EdgeImpulse & Sony

Overview

Automated machine learning (AutoML) is the process of applying machine learning (ML) models to real-world problems using automation. More specifically, it automates the selection, composition and parameterization of machine learning models. Automating the machine learning process makes it more user-friendly and often provides faster, more accurate outputs than hand-coded algorithms.

EdgeImpulse brings AutoML to Audio that simplifies each step in the machine learning process, from handling a raw dataset to deploying a practical machine learning model. In traditional machine learning, models are developed by hand, and each step in the process must be handled separately.

AutoML automatically locates and uses the optimal type of machine learning algorithm for a given task. It does this with two concepts:

• Neural architecture search, which automates the design of neural networks. This helps AutoML models discover new architectures for problems that require them.
• Transfer learning, in which pretrained models apply what they've learned to new data sets. Transfer learning helps AutoML apply existing architectures to new problems that require it.

Users with minimal machine learning and deep learning knowledge can then interface with the models through a relatively simple way using the EON Tuner tool developed by EdgeImpulse.

More specifically, here are some of the steps of the machine learning process that AutoML can automate, in the order they occur in the process:

• Raw data processing
• Feature engineering and feature selection
• Model andNeural Network Architectureselection
• Hyperparameter optimization and parameter optimization
• Deployment consideringInference time and on-device memory constraints
• Providing model Accuracy and Precision over Validation and Test Set, and on-device performance for your models before deploying

Processing input Raw Data via processing blocks require configuring lots and lots of Parameters. To start with Number of coefficients, Frame Length, Window size, FFT Length, Filter Number and the list never stops. While you've just started with Embedded ML, parameter selection might look daunting and that's the exact reason why EON Tuner exists. EdgeImpulse aims to make the process simple.

Why is EON Tuner Important?

EON Tuner is important because it represents a milestone in the fields of Embedded Machine Learning. Machine learning has been subject to the "black box" criticism -- meaning that machine learning algorithms can be difficult to reverse engineer, It can be difficult to track how the algorithm delivered that output. Consequently, this also makes it difficult to choose the correct model for a given problem, because it can be difficult to predict a result if a model is a black box.

EON Tuner helps to make machine learning less of a black box by making it more accessible. This process automates parts of the machine learning process that apply the algorithm to real-world scenarios.

Diving into the Details. Exploring EON Tuner:

To access EON Tuner, go the the annotated tab shown here. In this project we'll demonstrate how EON Tuner helps improve on-device performance and accuracy by optimizing and selecting and the most apt Model and Architecture based on your dataset. The selected dataset includes 4 classes of Audio data, including Audio inputs of Lions, bees, humans and gun shots all of which are threats to Elephants. Cumulatively around 7 minutes of data is collected.

Go ahead and head over to the EON Tuner section on EdgeImpulse Studio. You'll be prompted to a page where you'll have to configure target for your use-case.

"The EON™ Tuner will evaluate many candidate model architectures (selected based on your target device and latency requirements) concurrently to help you find the best performing architecture for your application." - EI Studio

Upon configuring target, you'll be prompted to EON Tuner Settings. Here, we can select the Dataset category for our Data type. Since this dataset includes Wildlife Audio Events, it lies perfect in Audible Events Category( one-time occurances of non-voice Audio ). Your use-case might also have a specific target, an MCU or or a specific Device Type where you'd have to deploy the ML Model. EON Tuner keeps that in mind and optimizes the model to run inference within the device constraints. For this a wide variety of targets and devices are mentioned to select from. We'll explore on-device performance comparing EON Tuner inference predictions to observe how robust this is! Given Target types include Arduino Nano Sense, ST IoT discovery kit, Silabs thunderboard, Sony's spresense, Eta Compute, and even Macbook processor. Here we'll optimize EON Tuner starting from Arduino Nano Sense up to couple of other boards.

Once you're done selecting configuration required for your use, go ahead and start training! A 7 minute dataset takes roughly 10 to 15 minutes to complete execution. You can have a snack in the meantime or resume your work and EdgeImpulse notifies you when the EON Tuner has completed execution via an email. Sometimes it can go upto 40 minutes or maybe even 2 hours depending on the size of your dataset.

So after the EON Tuner has completed training, your Dashboard should look something like this. Here, the selected target device is Arduino Nano 33 BLE Sense and the EON Tuner adjusts Performance metrics like Inference time and ROM/RAM usage based on the target device selected. The selected inference time threshold here is 500ms, so it lists all models which take upto 500ms to complete one inference and you can alter that for your use-case.

An impressive fact about EON Tuner: The dataset which earlier scaled upto 82.3% accuracy upon trying several different blocks and hits-and-trials, now reaches an astounding 92% accuracy!, 10% jump which means that in practical scenario, it would classify 10 more data samples accurately than the previous model. By default, the EdgeImpulse Studio recommends MFE or MFCC Processing block for Audio datasets, so for starters it takes some time to experiment and figure out which block is the best for the model.EON Tuner here, goes through all blocks and NN architectures and displays what's best for the dataset. For example - The spectrogram block performs the best here, but it took me some time and experimenting to figure out that. However EON Tuner did that in under 20 minutes, selecting what's best for the data.

Here you can see the change in accuracy, from 82.3% to 92% using the EON Tuner and how Spectrogram block proves to be more effective as compared to MFE. This is exactly how EON Tuner, AutoML in Audio works.

Sometimes, the slightest of change in parameters can make a big difference in the accuracy of the model. Each model takes in lots of parameters! Adjusting each one to the most optimum value is simply not possible and consumes a lot of time. Even though there are default values which yield a pretty decent accuracy, to achieve near-stellar accuracy, the parameters need to be adjusted just-right for the data. EON Tuner adjusts each one to the most apt value depending on the data.

The below example demonstrates how the slightest of change in input window size can cause a 3% change in accuracy.

With all the other input parameters same, and also the NN architecture to be same, a change in window size from 1000ms to 2000ms, brings in a large performance gap. Inference time increases from 157ms to 315ms, Neural Network size increases from 105kb to 207kb, and accuracy drops by 3%

Similarly, the EON Tuner provides more information regarding Parameters chosen as per the dataset like Frame stride, frame length, and number of frequency bands.

Similarly change in model architectures affect overall accuracy of the model as well. Here's comparison between a model having an extra dense layer and dropout layer compared to another one. The model with the dense layer performs 6% better than the other model with same input parameters. Such minute changes in NN Architecture may improve real-world performance.

Sometimes it might happen that retraining the model might cause increase/decrease in accuracy. ML isn't always predictable. So, I went ahead and retrained the model through EON Tuner to obtain 1% increase in overall accuracy, and much lower RAM (18kb DSP + 9kb NN) usage as well as faster inference( 239ms ) for the Arduino Nano Sense!! EON Tuner is powerful!

Here, MFE block performed really well as compared to Spectrogram block. Notable changes in NN Architecture as well as a few parameters are as follows -

Here, the Input Window size being 1000ms remains same as the previous one, but window increase is set to 500ms which affects the inference time in the model. This model has a significantly different model architecture too. It prefers to use 1D convolutional layer instead of 2D. Also, it opts for two dropout layers of rate 0.25 each as compared to one of 0.5 rate. This, overall alters the model performance. Also, makes it a bit lighter in size and speeds up the inference.

One of the most important features that EON Tuner supports is Filtering and Sorting through multiple models to pick which is the best. For eg, you can choose and filter amongst the DSP blocks. Let's say I prefer to use MFCC over MFE and Spectrogram, I can filter that block type and select the required model instead of searching through all types. Similarly, you can select the optimum Neural Network type 1D/2D. EON Tuner also allows viewing models ranked in terms of accuracy score over test or train instead of validation. One of the useful features is that you can sort between F1 or precision scores over a single class, (eg Bees) if that's more important over other classes. Additionally you can also sort between RAM/ROM/Latency scores.

This brings us to using the EON Tuner optimized model in our project. Head over to EON Tuner page and click "select" to select the model in your project. Here, I've already selected the one in the middle, and you could do the same by clicking select.

Now, once you've selected the model which fits best for the use, you can also explore around the parameters over the model, and also the different NN/Processing block versions.

This section allows you to manage multiple versions of the same block generated by EON Tuner. Here's how the current feature Explorer looks as compared to the previous one -

This shows a significant decrease in Incorrect Classifications by the EON Tuner optimized model as compared to the previous one ( on the right ). This provides an insight on which labels or data has the EON optimized model improved upon!

The Confusion matrix also shows an improve in classification as well as the class "lion" has better and accurate classification. You might observe a lot of changes and improvements in the EON Tuner model, I am highlighting the significant ones.

While most of the review has been covered let's move on too deploying the impulse on device and comparing expected inference time in-studio vs on-device.

For starters, you can try a couple of different things like Live Classification from the Studio or model testing on your test data as well. You can head over to "Live Classification" for that.

Deploying EON Tuner Model on-device:

I have a couple of different boards on my desk and I'll be testing the inference on a few of them -

All of these boards are officially supported by EdgeImpulse and have Audio built in.

To start off with, I'll test the model first on the ST-IoT Discovery kit and compare the inference time on-device with the studio predictions. This is essential to test how the EON Tuner optimizes and selects the best architecture while considering device and memory constraints.

1) Running Model on ST-IoT Discovery Kit:

The St-IoT Discovery kit is one of the very first boards to be supported by EdgeImpulse. More details on how to connect it with the studio here: https://docs.edgeimpulse.com/docs/unboxing-the-st-b-l475e-iot01a

Configuring Target on EON Tuner for the St-IoT discovery kit -

You can change the target device for EON Tuner anytime without the requirement of training the model all over again Just change the target device, and it'll display the adjusted device performance on the dashboard.

You might have noticed that the inference time and RAM usage has changed just a bit matching the device constraints. The total inference time has now been updated to 239 ms on device. Let's check it out! Head over to the deployment page and select the correct binaries for ST IoT discovery kit. After that flash the bin file to your board. Finally run, edge-impulse-run-impulse to see the classification output.

After you run the impulse, you'll see some Inference settings like Interval and frame size, sample length.

This is the expected output that you'll notice after running the model on the ST-IoT discovery kit. As you might notice, the inference time taken is 241ms + 12ms, which is very close to what was expected by the EON Tuner's on-device performance.

2) Running Model on Sony's Spresense:

Sony's Spresense is a small, but powerful development board with a 6 core Cortex-M4F microcontroller and integrated GPS, and a wide variety of add-on modules including an extension board with headphone jack, SD card slot and microphone pins, a camera board, a sensor board with accelerometer, pressure, and geomagnetism sensors, and Wi-Fi board - and it's fully supported by Edge Impulse. Read more here - https://docs.edgeimpulse.com/docs/sony-spresense

Configuring Target for Sony Spresense:

Select the Sony Spresense Board from the EON Tuner Settings page, and set it as target board. The Sony Spresense is packed with a massive 1536 kB RAM and 8192 kB ROM for Inference. See the fig below.

Our selected model uses solely 27kb RAM out of allocated 1536Kb RAM which Allows much faster and smoother performance on the board! The inference time is 239ms which is pretty standard and quick for acoustic models. Deploying the binary on the Sony Spresense board sees nearly same results, even better on-board which tests the efficacy of the board and system.

3) Running Model on Himax WE- I Plus:

Himax WE-I B is a powerful on-board processor, camera, audio and accelerometer. More about this board here: https://docs.edgeimpulse.com/docs/himax-we-i-plus

Configuring Target for Himax on EON Tuner:

So, Himax WE I Plus is packed with 2048 kb of RAM and ROM out of which it only uses around 20kb for the model, and uses around 239ms of inferencing time which is usually the same for all devices. Go ahead and build the binary in deployments tab and flash the board with the model to see it work!

Here's the classification output. So, Himax being a powerful device takes just 175ms + 6ms i.e. 181ms of inference time to run a classification and yet, classifies each class with decent accuracy, perfect for practical applications!

So that's all about it! Concluding, AutoML has vast benefits in the world of Embedded ML and simplifies the black box terminology in ML.

Here's the link to the project in case you want to test this out yourself and want to replicate results -

• Unoptimized project - https://studio.edgeimpulse.com/public/9621/latest
• EON AutoML optimized project - https://studio.edgeimpulse.com/public/41193/latest