Tiny ML Air Writing Recognition with Nicla Sense ME

Less than half a year ago, a new Arduino’s board, Nicla Sense ME (Motion & Environment), hit the market. And, needless to say, I couldn't wait for a chance to try this brand new device and explore its capabilities.

Nicla Sense ME is currently the smallest board across the entire Arduino PRO family of boards for industrial applications. It comes with a range of Bosch Sensortecsmart motion and environmental sensors, and I’m quite sure that this device will be widely applicable in industrial IoT, as it’s great for various research projects and commercial use.

Just have a look at its unique characteristics:

I’m really excited about the pace of development of the TinyML field because such small devices coupled with tiny machine learning capabilities indeed demonstrate enormous potential for many industries and domains.

Now, let me share my findings after an experiment with Nicla Sense ME.

Procedure:

In my experiment, I decided to play with in-air handwriting using an accelerometer-based pen device for handwriting recognition applications in order to test the ability of Nicla Sense ME to identify and process data about handwritten numbers on the edge. Compared to handwriting based on touch devices, this task is much more complex and challenging.

The device on which the experiment was carried out consisted of a triaxial accelerometer and Nicla Sense ME. The accelerations generated by hand motions were processed on MCU for further inference using the Neuton TinyML model.

Step 1: Model Training

For model training, I created my own dataset by capturing data containing 200 samples for each handwritten digit (from 0 to 9). Each sample was recorded for 2 seconds with a frequency of 100 Hz. It's easy to calculate how many times I had to move my hands in the air to collect this data, and it's even easier to imagine what my family members were thinking about me :)

Training Dataset

I merge captured samples to a single CSV file and add a target variable using Python script. The resultant dataset consisted of three features (accelerometer axes) and 400000 rows (200 samples by 200 rows for each digit).

I uploaded the dataset in a CSV format to the Neuton TinyML platform, selected the target variable (Digit), target metric (Accuracy). And then I enabled the TinyML option, selected 8-bit calculations without float support. For this case, I activated the Digital Signal Processing option for automatic data preprocessing and feature extraction. The model was trained automatically, and nothing needed to be compressed.

The platform allows us to see a preprocessed dataset. It merged 200 samples in one row and added some statistical features (min, max, mean, etc.)

I chose Neuton for this experiment since the platform automatically builds models that are optimal in size and accuracy. I also tried to train the same model with TensorFlow. Сheck out the comparison of the resultant metrics:

Just have a look at the dramatic difference in the number of coefficients, Neuton's model has 42 times fewer coefficients than a TensorFlow one!

Step 2: Embedding into a Microcontroller

After the training was completed, I downloaded the archive containing all the necessary files for the embedding to the microcontrollers firmware project.

I uploaded all the components to GitHub so that you can easily replicate my experiment for yourself: https://github.com/Neuton-tinyML/nicla_digits

Step 3: Running On-device Inferences

I connected my device to a PC via USB to display inference results. Check out the outcomes of my experiment on the video below:

As you can see, the sensor accuracy is really high so it managed to identify all the numbers with great probability. The model operates right on the device and weighs about 1 KB. That’s what I call real TinyML!

Conclusion:

I believe that the future belongs to such incredibly tiny machine learning solutions. The ability to sense and accurately process various types of data in real-time, at low energy consumption but with high computing power opens up new opportunities in many fields. The implementation of such solutions can, for instance, streamline robotic science, ease the detection of seismic activities and dangerous gases in the air, advance IoT devices that we use on a daily basis, and a lot more — the scenarios are infinite! Isn’t that inspiring?

Please write in the comments below what other experiments you want me to do with Nicla Sense ME!