TinyML: Training a Magic Wand

Begin personalizing your TinyML magic wand by training the model with data you collect.

IntermediateFull instructions provided1,730

Things used in this project

Hardware components

Arduino Nano 33 BLE Sense

USB-A to Micro-USB Cable

Software apps and online services

Arduino IDE

Jupyter Notebook

Text Editor

Story

1)Establishing the baseline for the project from the TinyML book

We first used the instructions from Pete Warden’s TinyML book as a starting point. We ran into issues uploading the code onto our Arduino Nano 33 BLE Sense board. The board would disconnect as soon as the code was uploaded. We realized the issue was a result of using the most recent version of the Arduino code, so we reverted to the version referenced in the book. Once we confirmed the code could be uploaded, we went in and patched the Arduino_LSM9DS1/src/LSM9DS1.cpp file (see picture) as directed in the book and then uploaded the code to the board.

2)Creating code to collect data from the accelerometer

Once we confirmed that the code was able to recognize all 3 gestures, we began collecting our own dataset for the 3 gestures. To do this, we used a “Accelerometer_DataCollection.ino” Arduino file that we created to collect and display the data in the correct format to replace the data previously used to train the model. To create this code, we referenced two external examples: Dale Giancono’s Nano33BLESensorExample_accelerometer.ino example, and code from Daniel Hertz on MakerPro. Both sources are referenced in our “Work Attribution” page.

3)Collecting our data

We each performed the gestures 10 different times while using the program located in the “Code” section to record the data from the accelerometer for a total of 30 data sets per gesture. We saved each data set as a plain text file.

4) Training a model from our data

To train the model, we used the code located at https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/micro/examples/magic_wand/train/train_magic_wand_model.ipynb. We created a copy of the Jupyter notebook and uploaded our data sets into the generated train directory alongside the pre-existing data. Then, we modified the names array in data_prepare.py to include the names on our plain text data files, and modified data_split_person.py so that one team member’s data each was used for training, verification, and testing. After running the notebook, we downloaded the generated model.cc file containing the new model. Our copy of the notebook is viewable at https://deepnote.com/project/a748e8ab-8c0c-4692-bf9c-dec9a33f7cd9.

5) Inputting the trained model into the existing framework

Next we opened up the C++ file, which contained two variables: the unsigned character array model_tflite[] and the unsigned integer model_tflite_len. We pasted their values into the magic_wand_model_data.cpp file, replacing the values of the constant unsigned character array DATA_ALIGN_ATTRIBUTE[] and the constant integer g_magic_wand_model_data_len respectively. The code uploaded successfully, but was giving the error “Didn’t find op for builtin opcode ‘RESHAPE’ version 1”. This was because the reshape opcode was not defined in the setup, so we had to add the following line to the Arduino file (magic_wand.ino):

86 micro_mutable_op_resolver.AddBuiltin(tflite::BuiltinOperator_RESHAPE, tflite::ops::micro::Register_RESHAPE());

After adding the new line, we uploaded the code to our board and the application worked properly.

Demo

Accelerometer Data Collection Code

/*
  Accelerometer_DataCollection.ino

  This code reads data from the Arduino Nano 33 BLE Sense's built-in 
  accelerometer and outputs it to Arduino's serial monitor. It cycles 
  through reading and displaying data for slightly less than 3 seconds,
  and delaying for exactly three seconds. It begins each new cycle of
  readings with spacing and the following characters "-,-,-" to 
  distinguish between the data points. This is the same format that
  the training code takes the data, so the output of the serial monitor
  can be copied directly into plaintext files to be read into the code 
  that trains the model.

  Much of this code, including the general stucture was taken from the 
  following source: 
  Nano33BLESensorExample_accelerometer.ino
  Copyright (c) 2020 Dale Giancono. All rights reserved..

  Other portions of this code were taken from Daniel Hertz from the 
  following article:
  https://maker.pro/arduino/tutorial/how-to-use-the-arduino-nano-33-bles
  -built-in-imu
*/

/*Include Header Files*/
#include "Arduino.h"
#include "Nano33BLEAccelerometer.h"
#include <Arduino_LSM9DS1.h>


/*Global Variables*/
//a counter to limit the number of cycles where values are displayed
int counter = 0; 


/*Setup Function*/
void setup()
{
    // Serial monitor setup
    Serial.begin(115200);
    while(!Serial);

    // Accelerometer setup
    Accelerometer.begin();
    Serial.println("-,-,-");    
}


/*Main loop*/
void loop()
{          
  // Initialize Variables
  float x, y, z = 0.05;

  // Check if the accelerometer is ready and if the loop has only
  //   been run less than 200 times (=~3 seconds displayed) 
  if (IMU.accelerationAvailable() and counter < 200)
  {
    // Read the accelerometer
    IMU.readAcceleration(x, y, z);
    // Scale up the values to better distinguish movements
    x = (int) (x*100);
    y = (int) (y*100);
    z = (int) (z*100);

    // Print the values to the Serial Monitor
    Serial.printf("%.1f,%.1f,%.1f\r\n", x, y, z);
    
  // When the loop has run 200 times, reset the counter and delay
  //   3 seconds, then print empty lines and the new datapoint sequence
  } else if (counter >= 200) {
    counter = 0;
    delay(3000);
    Serial.println("\n\n\n\n\n\n\n\n\n\n\n\n\n-,-,-");
  }
  // Increment the counter and delay .01 seconds
  counter += 1;
  delay(10);
}