TinyML Made Easy: Gesture Recognition

TinyML

This project mixes Machine Learning (that is part of Artificial Intelligence) with a small device (Wio Terminal), which is nothing more than a microcontroller and sensors, whose main characteristics are ultra-low power consumption, 32-bit CPU, and a few kilobytes of memory. This new field of engineering is known as Embedded Machine Learning, or tinyML.

As we know, Microcontrollers (or MCUs) are very cheap electronic components, usually with just a few kilobytes of RAM, designed to use tiny amounts of energy. Nowadays, MCUs can be found embedded in almost any consumer, medical, automotive, and industrial devices. It is estimated that over 40 billion microcontrollers are sold every year, and probably hundreds of billions of them are in service nowadays. But, interestingly, those devices don’t get much attention because they’re often only used to replace functionality that older electro-mechanical systems could do in cars, washing machines, or remote controls.

More recently, with the IoT (Internet of Things) era, a significant part of those MCUs is generating “quintillions” of data, that in its majority, is not used due to the high cost and complexity (bandwidth and latency) of data transmission.

On the other side, in recent decades, we have seen a lot of development of Machine Learning models (aka Artificial Intelligence) trained with tons of data in very powerful and hungry mainframes.

But what is happening today is that suddenly, it become possible to take noisy signals like images, audio, or accelerometers and extract meaning from them by using neural networks. And what is more important is that we can run these networks on microcontrollers and sensors themselves using little power, interpreting much more of those sensor data that we are currently ignoring. This is tinyML, a new area that enables machine intelligence right next to the physical world.

The novelty area of tinyML can help bring good to our society.

The Wio Terminal

The Wio Terminal, a very affordable $36 device, is using an ATSAMD51P19 microcontroller with ARM Cortex-M4F running at 120MHz (boost up to 200MHz), 4MB of external flash memory, and 192KB of RAM. Wireless connectivity with Realtek RTL8720DN support. It is compatible with Arduino and MicroPython. It supports both Bluetooth and Wi-Fi providing a solid foundation for IoT and tinyML projects. There is a 2.4- inch LCD screen on Wio Terminal, an onboard IMU (LIS3DHTR), microphone, buzzer, microSD card slot, light sensor, and IR emitter (IR 940nm). Most importantly there are two multi-functional Grove ports onboard for the Grove ecosystem and Raspberry Pi compatible 40-pin GPIO pins for additional add-on support.

Wio Terminal Programming (Codecraft)

The easiest way to start with the Wio is using CodeCraft, a graphical programming platform based on Scratch3.0 that supports both graphical and text programming languages. Powered by Edge Impulse, Codecraft provides a beginner-friendly (yet powerful) web interface and toolkit for the whole tinyMl pipeline starting from data collection all the way up to model deployment and extensive use.

For starting, open Codecraft page and enter (or create) your credentials:

Now select Wio Terminal as your hardware for programming:

Wio Terminal - USB Connection

Programming the Wio Terminal via a personal computer is very simple. You should only connect the terminal to your computer via a USB cable, as shown below:

Now that you have already connected the Wio physically to your computer via USB and, you should also connect it with Codecraft via a serial port. For that, press "Connect" on Codecraft control panel:

If it is your first time doing the connection, you should download and install a Device Assistant:

Next, the Serial Port where the Wio is connected should be listed. Press Connect:

If everything is correct, a connection symbol should appear next to Wio Terminal Icon:

And note that using Device Mode, several programming blocks can be used. For example, in System, you can find the Arduino Main Procedure Block:

and, important blocks related with Wio sensors as 3 Axis Accelerometer, Light, Buttons, and Sound and, also actuators as a speaker and Infrared emitter:

Another very important actuator is the Display, which has a specific menu of blocks for it.

A complete guide for Codecraft can be found on Guide for Codecraft using Arduino.

Testing the Wio with a special Blink

Now that the Wio is properly connected, let's test it with a simple "Blink" program, but instead of a LED, we will use the display to blink between 2 different colors.

All programs start with Arduino Main Procedure Block, which has 2 parts: a) the Setup that will be executed only once and b) the Loop, which will be executed continuously. In this case, we want that the display to continuously alternate its color between red and blue. So, we will use 2 Display blocks that change the screen background color each 1 second. For timing, we should use a Delay ms block that can be found under Control.

At this point, you have your code ready. To see the results on the Wio, you should upload it to the terminal:

The Wio will continuously change its color:

Testing Buttons

Let's explore a little bit more the Wio, testing its user programmed buttons (there are 3, named A, B and C at top of the terminal). Now, during Setup, we will write a text, that will appear only once: "Press a button to select color display". And, during Loop, for each one of those buttons we will select a different display color:

Here the result:

As a side comment, all code that you create with blocks, automatically generates an Arduino compatible code, that can be analyzed as shown below. This is a great tool to help you to learn C/C++ programming.

Testing the Accelerometers

For our Gesture Recognition project, the most important sensor that will be used is the internal Accelerometer. The Wio has a 3-Axis Accelerometer, the LIS3DHTR.

The accelerometers commonly used in mobile phones are MEMS (Microelectromechanical) accelerometers. The module used in Wio Terminal is called 3-Axis Digital Accelerometer (LIS3DHTR). Generally, the internal structure of accelerometers consists of Capacitive Plates. Some types of accelerometers use fixed capacitive plates, while some of them have the plates attached to minuscule springs that move internally depending upon the acceleration forces acting on the sensor.

To know more details about Wio Terminal, read the document No-code Programming to Get Started with TinyM

Below is a simple code that will display the 3-Axis accelerometers values on Wio Terminal:

You can also see the values printed on Serial Monitor. For that, you should initialize the Serial Port and use the Serial Printf block:

TinyML - Gesture Recognition project

Now that we know the basics of how to handle the Wio Terminal and Codecraft, it is time to create a complete Embedded Machine Learning Project.

Let's follow this Machine Learning basic Workflow:

Decide a Goal (Model Creation)

Our project will classify 3 different gestures: Left-Right, Up-Down, and Idle:

This is a project related to Motion Recognition, so let's select this type of model and give a name for it:

Collect a dataset (Data Acquisition)

The next step is to collect our dataset. For that, we should first define the data labels. We do this, changing the LABEL names that are on the right of Codecraft. Let's use:

• left-right
• up-down
• idle

Once the new names are defined:

we can define on the blocks, what data should be collected when a button is pressed.

Once uploaded the Data Acquisition code to Wio, every time that a specific button is pressed, 5 seconds of data is collected and shown on the right panel:

We should repeat this procedure for all data labels. For Idle I also collected some data with the Wio over the table.

For this project, we will collect 50 seconds of each one of them.

Here one of the samples collected as shown in the next section

Training the Model

Once we have the dataset collected, we should pre-process the data and define what will be our Model Architecture (and its hyper-parameters) to be used during Training. The options of Codecraft are 3 models: DNN (Small and Medium) and CNN (Large). We will choose Medium, where the raw time-series data is converted to a small number of parameters, by using FFT technics (Fast Fourier Transformer) and feed to a DNN (Dense Neural Network model), with 20 and 10 neurons on hidden layers.

And we will keep the suggested model hyper-parameters:

Now, you should Start Training:

Once the training ended (you can follow it on the Log tab), go to Model Training Report and check the model Accuracy and Confusion Matrix. If the Accuracy is low (or the Loss is high), you should return to Model design and change it or try other hyper-parameters. In our case here, the model is simple and we will get around 100% of accuracy, which is very good (but difficult to achieve with more complex and real machine learning projects).

The next step will be to generate the final model to be deployed on our Wio Terminal. We do it by choosing the final model parameters type (int8 or float32) and pressing the button Model Deployment. We will keep the int8 trained model, where the parameters are integer bytes, which makes the model smaller.

The final model (Use Model) can be found on programming section:

Making Inference

The last, but not least step of our tinyML project is to use the model with new unseen data (Inference).

Let's create a simple code that during the Loop, will get data from the 3-Axis accelerometer and feed it to our trained model to perform classification. We will print the output values of the model on Serial Monitor and show the current prediction on Wio Display:

Here is a snapshot of the result (that was very good!):

As a last comment, it is always important to remember that the final code can also be generated and saved to be used on an Arduino IDE, for example.

Conclusion

That's it! It is really very simple to create a tinyML project using Wio Terminal and the Codecraft IDE powered by Edge Impulse!

Please visit my blog for other projects: https://mjrobot.org/

See you on my next tutorial!

Saludos from the south of the world!

Marcelo Rovai