Build a Smart Dumbbell with the SAMD21 Machine Learning EVK

Objective

Turn your dumbbell into a smartbell with the SAMD21 Machine Learning Evaluation Kit and an embedded ML classifier built in the Edge Impulse Studio.

With the Edge Impulse, it is now is easier than ever to develop embedded machine learning solutions that can be deployed on Microchip Arm® Cortex®-based 32-bit microcontrollers and microprocessors. This example will guide you through the process of:

• Collecting 6-axis IMU data from the SAMD21 ML Evaluation Kit
• Uploading the collected data to the Edge Impulse Studio
• Creating a custom impulse that can classify the input data
• Deploying the impulse back to the SAMD21 firmware

See the bullets below to get an idea of the size and performance of the smartbell application.

• Build Parameters: -O2 optimizations, 48MHz Clock
• ROM Size: ~104kB
• RAM Size: ~9kB
• Inference Time: ~379ms (DSP: 369ms, NN: 1ms, Anomaly: 9ms)

Connection Diagram

SAMD21 ML Evaluation Kit with Bosch IMU

The IMU 2 click board is connected to the mikro BUS socket. Check the pin labels when connecting the boards to ensure that they are connected properly.

Dumbbell Mounting

The kit is mounted on one end of the dumbbell using double sided sticky tape and rubber bands. The USB cable is tied around the handle to prevent the kit from being pulled off.

Procedure

Before getting started, be sure to install all the necessary software tools. We will work within MPLAB X and build the project with the XC32/32++ Compiler. Both MPLAB Data Visualizer and the Machine Learning Plugin are available within the plugin update center in MPLAB X. You will also need to sign up for a free account on the Edge Impulse Studio and then create a new project so that you can begin uploading training data.

This guide uses the Bosch version (IMU2) of the SAMD21 ML Kit. For the TDK version (IMU14), use the SAMD21_ML_Kit_Datastreamer.X firmware to collect and upload data to the Edge Impulse Studio.

Data Collection

The first step in building a machine learning model is collecting data. This data will be used to train the ML model so that it can recognize different types of exercises. In this example we will train the model to recognize three exercises.

To begin collecting data, the SAMD21 ML Kit must be programmed with the data streamer firmware. Download the example project (Smartbell-SAMD21-BMI160.X) and open it in MPLAB X. Check that the DATA_STREAMER_BUILD flag is set to 1, and then click Make and Program Device.

Next, open MPLAB Data Visualizer and load the workspace found within the downloaded example project (mplab-dv-workspace-6dof-imu-data.json). Be sure that the DGI connection for the SAMD21-IoT WG is disabled. Select the appropriate Serial/CDC Connection for the SAMD21, adjust the baud rate to 115200, and then click Apply. To start parsing the serial data, click the Play button.

Navigate to the Variable Streamers tab and select the Serial/CDC Connection as input for the variable streamer. This will parse the six axes of IMU data from the UART stream.

You should now see the 6-axis IMU data plotted in the graph.

Now that the data is available in MPLAB Data Visualizer, you are ready to start collecting samples to train a machine learning model. It is best to have at least a few minutes of data for each type of exercise. Begin by performing a few repetitions of one exercise, and then double click the graph in the Time Plot to stop it from scrolling. Fit the region of data corresponding to the exercise within the graph window and then click Mark. This will set the cursors at the boundaries of the window to mark that region for upload to the Edge Impulse Studio.

Next, log in with your Edge Impulse credentials within the ML Plugin to configure the data upload. Select ax, ay, az, gx, gy, and gz as the Data Sources. In the Project field, select the Edge Impulse Studio project that the data will be uploaded to, and select Training as the Data Endpoint.

When configuration is complete, click Upload Data, and then open the Edge Impulse Studio within your browser. Go to the Data acquisition tab to see the uploaded data.

Sensor names should remain consistent within each project in the Edge Impulse Studio. If you are adding more data to a project that already contains data, then be sure to use the same sensor names.

Follow this process for uploading data samples of each exercise until you have at least a few minutes of data for each. The samples can be of any length because the Edge Impulse Studio will handle slicing the data into windows of equal size. Collect data of each exercise performed by the right arm and the left arm. The model can only learn from what it will see in the training data, so it is important to build a data set that is representative of the actual data that the model will see in deployment. If the data set only contains data from one person, the model will learn the exact movement patterns of that person. If the data set contains data from many people, the model will learn the general characteristics of each exercise that exist across different people.

Impulse Creation

Once you have a training data set, navigate to the Impulse design tab where you can configure the window size of the data input into the impulse and select the processing blocks and learning blocks that the impulse will include. For this continuous motion recognition problem we will use a Spectral Analysis block, a Neural Network block, and an Anomaly Detection block. Once the blocks have been added and the time series data has been configured, click the Save Impulse button. Start with a window size of 2000 ms and a window increase of 250 ms.

Time series data

The first block in the impulse is always the input data. The Edge Impulse Studio uses a sliding window to go over the raw data and slice it into samples of equal length. The Window increase is the offset used to shift the window between each snapshot. When the Window increase is less than the Window size, the sliding window will create a larger number of samples that overlap somewhat yet are still unique.

The Window size setting will depend on the length of the event that the model will learn to recognize. For periodic motion data like repetitions of a dumbbell exercise, the window size should include at least one complete performance of the exercise. It is a design trade-off between speed and accuracy. If the model is performing very well, then it may be possible to decrease the window size and thereby decrease the latency of the system while maintaining an acceptable level of accuracy.

Spectral Analysis

The data samples sliced by the sliding window will be passed into the Spectral Analysis block for pre-processing and feature extraction. The extracted features will be the actual data input into the learning blocks. Open the Spectral Features tab to configure the data pre-processing parameters.

The Spectral Analysis block allows for scaling, filtering, and spectral power analysis. The graph at the top shows the loaded data file. Drag the window to inspect different sample windows within the file. The graphs on the right show output of the Spectral Analysis block to help with parameter tuning.

For the smartbell example project, the parameters are configured as shown in the above image. When you are ready to move on to the next step, click Save Parameters.

Next, click Generate Features and use the feature explorer to inspect the generated features. Each data sample will be colored in the graph according to its' label. Use the drop-down box for each axis to plot the samples based on the different features. It is a good sign when the data of different labels are well separated in multiple dimensions.

Neural Network (Keras)

The neural network classifies each data sample by outputting a confidence score for each of the labels present in the training data set. The input to the network is the feature array that is output from the Spectral Analysis block. The network type and structure can be configured in the NN Classifier menu.

For this example project the Number of learning cycles was set to 22 and the Learning rate was set to 0.001. The general rule in adjusting these two parameters is to find a sweet-spot where the model achieves a high accuracy by steadily improving with each training cycle. The Minimum confidence rating is simply a threshold for trusting the neural network result.

The hidden layers of the network can be edited in the Neural network architecture menu. There are many types of layers available, but for this example all that is required are two fully connected dense layers. The latency and memory requirements can be minimized by selecting the simplest model that provides acceptable accuracy.

The Edge Impulse Studio enables you to experiment with different network architectures and quickly evaluate their performance in order to select the optimal network for any particular problem. When the network is performing well, you can move on to training the anomaly detection block.

K-Means Anomaly Detection

The K-Means Anomaly Detection layer is used with in parallel with the neural network classifier in order to detect data that is significantly different from data found in the training data set. This is generally useful as an indication of the reliability of the neural network. In this example it will also serve as an indication of the correctness of the exercise being performed with the smartbell. For instance, if the exercise is performed too fast or with incorrect form then the sample would be flagged as anomalous.

The k-means parameters can be adjusted in the Anomaly detection settings and the axes that will be used as input can be selected as well. The RMS values for each axis are the recommended input when using the Spectral Analysis block. The cluster count parameter can be increased if the distribution of data across the generated features is more complex, so that more clusters can be used to approximate the distribution.

After the K-Means Anomaly Detection block is trained you can move on to testing the impulse performance.

Impulse Testing

Once the neural network and anomaly detection layers have been trained, the performance can be tested by uploading new data to the testing endpoint. The same data upload process is used except that the Data Endpoint must be changed to Testing. Testing data is kept separate from training data so that the model performance can be verified with data that the model has not seen during training.

Live Classification

If you have the Live Classification tab open in the Edge Impulse Studio, it will automatically display the classification results for new data uploaded to the testing endpoint. Live Classification gives a detailed classification result for each sample and provides tools for inspecting each tested data sample within the data set.

The Live Classification tool will slice uploaded data into sample windows using the window size and offset from the Impulse Design step. Then the sample windows are passed through the impulse for classification.

Model Testing

The Model Testing tool can be used to test model performance over the entire test data set or over batches of test data as needed. It gives an accuracy score based on the expected outcome for each test data file.

Model testing is useful for scoring overall performance of the impulse and for identifying any samples that might require a closer look in the Live Classification tool.

Impulse Deployment

Once the impulse design and testing is complete, go to the Deployment tab to download the source code that can be added to your MPLAB X project. Select the C++ Library option and then scroll down to select optimizations.

The Edge Impulse EON compiler greatly reduces the memory requirements and has no negative impacts on performance, so it is activated by default. There is also an option to quantize the NN to 8-bit values, which can save some ROM at the expense of RAM. For this example 8-bit quantization is used because the difference in memory usage is trivial and the impulse actually has a slightly higher performance when quantized. When the desired optimizations are selected, click Build to download the source code for the impulse.

Firmware Integration

Open the src folder within the example firmware, Smartbell-Edge-Impulse.X, and then replace the model-parameters and tflite-model folders with the folders of the same name from the newly downloaded source code. Set DATA_STREAMER_BUILD equal to 0 at the top of main.cpp and then program the SAMD21 ML Kit.

Viewing Embedded Classification Results

Once the device is programmed, leave it connected to the PC via USB and open MPLAB Data Visualizer for displaying the classification results. Select the appropriate Serial/CDC Connection for the SAMD21, adjust the baud rate to 115200, and then click Apply. Click play on the configured Serial/CDC Connection, and then open the Terminal and select the Serial/CDC Connection from the drop-down box.

The impulse deployment firmware collects IMU data in two second windows and then passes each window of data through the impulse for classification. The classification results are then transmitted over UART for display in the terminal.

Conclusions

Now that you have seen how easy it is to build and deploy an embedded machine learning classifier with the SAMD21 ML Evaluation Kit and Edge Impulse, you are ready to add embedded classification and anomaly detection to your next application. The SAMD21 ML Kit can support a variety of Sensor Click Boards from MikroE, so get creative and explore the possibilities of embedded ML with Edge Impulse.