Hardware components | ||||||
 |
| × | 1 | |||
 |
| × | 1 | |||
| × | 1 | ||||
| × | 1 | ||||
| × | 1 | ||||
| × | 1 | ||||
| × | 1 | ||||
| × | 1 | ||||
Software apps and online services | ||||||
 |
| |||||
 |
| |||||
| ||||||
 |
| |||||
| ||||||
 |
| |||||
Hand tools and fabrication machines | ||||||
 |
| |||||
Like many of us, I spent most of my waking hours on the computer while working or studying. However, recently, I have started to feel a precipitous pain on my forearm after utilizing the mouse longer than six hours. Therefore, I decided to create this project to get prescient warnings regarding forearm muscle soreness levels so as to prevent any injury risks.
Since utilizing the mouse involves monotonous movements of the same small muscle groups for protracted periods of time, overusing the mouse by positioning, traveling, scrolling, and clicking can make these muscle groups tired and overworked. Hence, after repetitive overexposure to monotonous mouse movements, muscle stress (soreness) increases precipitously and can engender[1]:
- Pain (ache, soreness) on the top of the hand
- Pain (ache, soreness) around the wrist
- Pain (ache, soreness) along the forearm and the elbow
- Numbness and tingling in the thumb and the index finger
- Burning, stiffness, restricted range of motion
Also, it can cause soreness and fatigue by putting extra strain on the muscles in the upper back (trapezius muscle) and shoulder (deltoid muscle).
Generally, the aforementioned symptoms are temporary and get better on their own after reducing mouse use. Nevertheless, overexposure to monotonous mouse movements can cause more noxious and enervating disorders such as Repetitive strain injury (RSI) and Carpal tunnel syndrome (CTS).
Repetitive strain injury (RSI) results from forceful, awkward, and/or repetitive use of your limbs, producing damaged muscles, tendons, and nerves. Although RSI is a broad term that encompasses several disorders, general symptoms include tingling or loss of sensation in fingers, inability to grasp objects between thumb and fingers, decrease in the size of hand muscles, and pain in the wrist, elbow, shoulder, or neck. The severity of RSI cases varies widely. Tendonitis is the most common example of RSI, while Carpal tunnel syndrome is a more rare and serious disorder.
Carpal tunnel syndrome (CTS) and Thoracic outlet syndrome (TOS) are two of the most disabling repetitive strain injuries. These conditions are disorders of the tendons, nerves, arteries, or veins, occurring at the wrist and upper arm, respectively. In CTS, repeated bending or use of the wrist and fingers results in the compression of the median nerve (runs along the palm side of the wrist), causing intermittent numbness, tingling, and pain in the side of the hand, including the thumb through the inside of the ring finger[2].
After scrutinizing recent research papers on mouse fatigue, I decided to utilize GSR (galvanic skin response) and EMG (Electromyography) measurements denoting forearm muscle soreness and create a budget-friendly device to forecast muscle soreness levels in the hope of averting permanent mouse-related injuries.
Since muscle soreness levels depending on GSR and EMG measurements fluctuates for different muscle groups, it is not possible to extrapolate and interpret forearm muscle soreness levels by merely employing limited data without applying complex algorithms. Therefore, I decided to build and train an artificial neural network model by utilizing the empirically assigned soreness classes to predict forearm muscle soreness levels based on GSR and EMG measurements.
I decided to employ Wio Terminal in this project since it can easily collect muscle soreness data (GSR and EMG measurements) and run my neural network model after being trained to forecast forearm muscle soreness levels. To obtain the required data to train my model, I connected a GSR sensor (Grove) and an EMG detector (Grove) to Wio Terminal. Also, Wio Terminal can display the collected muscle soreness data on its integrated TFT LCD screen.
Since Wio Terminal supports reading and writing information from/to files on an SD card, I stored the collected data in a CSV file on the SD card to create a data set. In this regard, although Wio Terminal provides Wi-Fi connectivity, I was able to save data packets via Wio Terminal without requiring any additional procedures.
After completing my data set, I built my artificial neural network model (ANN) with TensorFlow to make predictions on forearm muscle soreness levels (classes) based on GSR and EMG measurements. As labels, I employed the empirically assigned muscle soreness classes for each data record while collecting data:
- Relaxed
- Tense
- Exhausted
After training and testing my neural network model, I converted it from a TensorFlow Keras H5 model to a C array (.h file) to execute the model on Wio Terminal. Therefore, the device is capable of detecting precise forearm muscle soreness levels (classes) by running the model independently.
Lastly, to make the device as compact and robust as possible while operating, I designed a Pikachu-inspired case (3D printable).
So, this is my project in a nutshell 😃
In the following steps, you can find more detailed information on coding, logging data on the SD card, building an artificial neural network model with TensorFlow, and running it on Wio Terminal.
🎁🎨 Huge thanks to Seeed Studio for sponsoring these products:
⭐ Wio Terminal | Inspect
⭐ Grove - GSR Sensor | Inspect
⭐ Grove - EMG Detector | Inspect
🎁🎨 Also, huge thanks to Creality3D for sponsoring a Creality CR-6 SE 3D Printer.
🎁🎨 If you want to purchase some products from Creality3D, you can use my 10% discount coupon (Aktar10) even for their new and most popular printers: CR-10 Smart,CR-30 3DPrintMill,Ender-3 Pro, and Ender-3 V2.
🎁🎨 You can also use the coupon for Creality filaments, such as Upgraded PLA (200g x 5 Pack),PLA White, and PLA Black.
Since I wanted to collect forearm muscle soreness data and run the model while working or studying, I decided to design a Pikachu-inspired case to create a compact and sturdy device operating flawlessly on my desk. I thought adding Pikachu to my case would be funny due to Pikachu's notorious Thunder Shock attack stunning the opponent's muscles.
I designed the main case and its back cover in Autodesk Fusion 360. You can download their STL files below.
For the Pikachu affixed to the main case, I utilized this model from Thingiverse:
Then, I sliced 3D models (STL files) in Ultimaker Cura.
Since I wanted to create a solid structure for the case and apply a stylish theme to the device, I utilized these PLA filaments:
- Purple
- Yellow
Finally, I printed all parts (models) with my Creality CR-6 SE 3D Printer. Although I am a novice in 3D printing and it is my first FDM 3D printer, I got incredible results effortlessly with the CR-6 SE :)
// Connections
// Wio Terminal :
// Grove - GSR sensor
// A0 --------------------------- Grove Connector
// Grove - EMG Detector
// A2 --------------------------- Grove ConnectorTo collect and display forearm muscle soreness data, I connected the GSR sensor (Grove) and the EMG detector (Grove) to Wio Terminal. Since Wio Terminal has one Grove port supporting analog sensors, I connected the EMG (Electromyography) detector via a Grove male jumper conversion cable, as shown in the schematics below.
GSR (galvanic skin response) sensor needs to be calibrated by adjusting the integrated resistor until the sensor generates 512 as the output signal on the serial monitor before wearing it.
You can get more information regarding coding in Step 3.
After printing all parts (models) and completing sensor connections successfully, I fastened all components to the main case and made connection points rigid by utilizing screws and a hot glue gun.
Finally, I attached the back cover to the main case via screws and affixed Pikachu to the top via the hot glue gun.
Since Wio Terminal supports reading and writing information from/to files on an SD card, I decided to utilize a CSV file on the SD card so as to log the collected forearm muscle soreness data without applying any additional procedures. However, before proceeding with the following steps, I needed to set up Wio Terminal on the Arduino IDE and install the required libraries for this project.
#️⃣ To set up the Seeed SAMD Arduino Core, open the Arduino IDE, click Tools ➡ Board ➡ Boards Manager, and search for Wio Terminal in the search box. Then, install Seeed SAMD Boards.
#️⃣ Download the required libraries for Wio Terminal:
Seeed_Arduino_FS | Download
Seeed_Arduino_Linechart | Download
Step 2.1: Loading and displaying images from the SD cardTo display images on the TFT LCD screen integrated into Wio Terminal successfully, I needed to convert them to a compatible BMP file format and then load them from the SD card.
#️⃣ First, open Microsoft Paint to rescale images to the required sizes and save them as the 24-bit bitmap (.bmp) files in the bmp folder.
#️⃣ To convert the 24-bit bitmap (.bmp) files to Wio Terminal's required BMP file format, download the bmp_converter.py file and save it to the bmp folder.
#️⃣ Then, modify the folder location in the bmp_converter.py file and run it.
#️⃣ Enter 1 for 8-bit BMP file format conversion or 2 for 16-bit BMP file format conversion.
#️⃣ Finally, the bmp_converter.py file converts all the given 24-bit bitmap (.bmp) files and saves them to rgb332 (8-bit) or rgb565 (16-bit) folders in the bmp folder.
#️⃣ To display the converted BMP files on the TFT LCD screen, move them to the SD card.
#️⃣ Then, copy the RawImage.h file to the sketches on the Arduino IDE.
// To draw the 8-bit color image on screen, starting from point (x, y):
drawImage<uint8_t>("path to sd card image", x, y);
// To draw the 16-bit color image on screen, starting from point (x, y):
drawImage<uint16_t>("path to sd card image", x, y);After setting up Wio Terminal and installing the required libraries and modules, I programmed Wio Terminal to collect GSR and EMG measurements and save them to the given CSV file on the SD card.
Since I needed to assign forearm muscle soreness levels (classes) empirically as labels for each data record to create a valid data set, I utilized the configurable buttons integrated into Wio Terminal to choose among muscle soreness classes. After selecting a muscle soreness class, Wio Terminal adds the selected class to the collected muscle soreness data and then saves the data record to the given CSV file on the SD card as a new row.
- Button A ➡ Relaxed
- Button B ➡ Tense
- Button C ➡ Exhausted
You can download the mouse_fatigue_detection_data_collect.ino file to try and inspect the code for collecting forearm muscle soreness data and saving information to a given CSV file on the SD card.
⭐ Include the required libraries.
#include <SPI.h>
#include <Seeed_FS.h>
#include "TFT_eSPI.h"
#include "seeed_line_chart.h"
#include "SD/Seeed_SD.h"
#include "RawImage.h"⭐ Define the TFT screen and the sprite settings.
TFT_eSPI tft;
// Define the sprite settings:
#define max_size 50 // maximum size of data
doubles gsr_data, emg_data;
TFT_eSprite spr = TFT_eSprite(&tft);⭐ Initialize the File class and define the CSV file name.
File myFile;
const char* data_file = "mouse_fatigue_data_set.csv";⭐ Check the connection status between Wio Terminal and the SD card.
⭐ Initiate the TFT screen. Then, create the sprite.
if(!SD.begin(SDCARD_SS_PIN, SDCARD_SPI)) while (1);
// Initiate the TFT screen:
tft.begin();
tft.setRotation(3);
tft.fillScreen(background_color);
tft.setTextColor(text_color);
tft.setTextSize(2);
// Create the sprite.
spr.createSprite(TFT_HEIGHT / 2, TFT_WIDTH);⭐ Define and display the required 16-bit images saved on the SD card.
drawImage<uint16_t>("data_collect.bmp", TFT_HEIGHT, 0);
drawImage<uint16_t>("carpal_tunnel.bmp", TFT_HEIGHT/2, 0);
drawImage<uint16_t>("mouse.bmp", TFT_HEIGHT/2, TFT_WIDTH-90);⭐ In the get_GSR_data function, calculate the average of the last ten GSR sensor measurements to remove the glitch.
⭐ Also, add a calibration value if necessary to rectify the generated results.
void get_GSR_data(int calibration){
long sum = 0;
// Calculate the average of the last ten GSR sensor measurements to remove the glitch.
for(int i=0;i<10;i++){
sum += analogRead(GSR);
delay(5);
}
gsr_value = (sum / 10) - calibration;
Serial.print("GSR Value => "); Serial.println(gsr_value);
}⭐ In the get_EMG_data function, evaluate the summation of the last 32 EMG sensor measurements.
⭐ Then, shift the summation by five with the right shift operator (>>) to obtain the correct EMG value.
void get_EMG_data(){
long sum = 0;
// Evaluate the summation of the last 32 EMG sensor measurements.
for(int i=0;i<32;i++){
sum += analogRead(EMG);
}
// Shift the summation by five with the right shift operator (>>) to obtain the EMG value.
emg_value = sum >> 5;
Serial.print("EMG Value => "); Serial.println(emg_value); Serial.println();
delay(10);
}⭐ Obtain current measurements generated by the GSR sensor and the EMG detector.
⭐ Initialize the sprite.
get_GSR_data(3);
get_EMG_data();
// Initialize the sprite.
spr.fillSprite(background_color);⭐ Adjust the line chart data arrays.
if(gsr_data.size() == max_size) gsr_data.pop();
if(emg_data.size() == max_size) emg_data.pop();⭐ Then, append new data variables to the line chart data arrays.
gsr_data.push(gsr_value);
emg_data.push(emg_value);⭐ In the display_line_chart function, define the line graph title (header) settings and draw it on the sprite.
⭐ Then, define the line chart's customizable appearance settings and draw the line chart on the sprite.
void display_line_chart(int header_y, const char* header_title, int chart_width, int chart_height, doubles data, uint32_t graph_color, uint32_t line_color){
// Define the line graph title settings:
auto header = text(0, header_y)
.value(header_title)
.align(center)
.valign(vcenter)
.width(chart_width)
.color(tft.color565(243,208,296))
.thickness(2);
// Define the header height and draw the line graph title.
header.height(header.font_height() * 2);
header.draw();
// Define the line chart settings:
auto content = line_chart(0, header.height() + header_y);
content
.height(chart_height) // the actual height of the line chart
.width(chart_width) // the actual width of the line chart
.based_on(0.0) // the starting point of the y-axis must be float
.show_circle(false) // drawing a circle at each point, default is on
.value(data) // passing the given data array to the line graph
.color(line_color) // setting the line color
.x_role_color(graph_color) // setting the line graph color
.x_tick_color(graph_color)
.y_role_color(graph_color)
.y_tick_color(graph_color)
.draw();
}⭐ Display the line charts with the given appearance settings on the TFT screen.
display_line_chart(0, "GSR", TFT_HEIGHT/2, 90, gsr_data, text_color, tft.color565(165,40,44));
display_line_chart(110, "EMG", TFT_HEIGHT/2, 90, emg_data, text_color, tft.color565(165,40,44));
spr.pushSprite(0, 0);
spr.setTextColor(text_color);
delay(50);⭐ In the save_data_to_SD_Card function, open the given CSV file on the SD card in the APPEND file mode.
There are three file modes supported by Wio Terminal: WRITE, READ, and APPEND.
⭐ If the given CSV file is opened successfully, create the data record including the selected muscle soreness class to be inserted as a new row.
⭐ Then, append the data record as a new row and close the CSV file.
⭐ After appending the given data record successfully, notify the user by displaying the selected muscle soreness class on the TFT screen.
⭐ If Wio Terminal cannot open the given CSV file successfully, print this message on the TFT screen: Wio Terminal cannot open the file!
void save_data_to_SD_Card(String Soreness){
// Open the given CSV file on the SD card in the APPEND file mode.
// FILE MODES: WRITE, READ, APPEND
myFile = SD.open(data_file, FILE_APPEND);
// If the given file is opened successfully:
if(myFile){
Serial.print("Writing to "); Serial.print(data_file); Serial.println("...");
// Create the data record to be inserted as a new row:
String data_record = String(gsr_value) + "," + String(emg_value) + "," + Soreness;
// Append the data record:
myFile.println(data_record);
// Close the CSV file:
myFile.close();
Serial.println("Data saved successfully!\n");
// Notify the user after appending the given data record successfully.
tft.fillScreen(background_color);
drawImage<uint16_t>("data_collect.bmp", TFT_HEIGHT/4, 0);
tft.drawString("Selected Soreness Class: " + Soreness, 0, 140);
tft.drawString("Data Stored!", 86, 180);
}else{
// If Wio Terminal cannot open the given CSV file successfully:
Serial.println("Wio Terminal cannot open the given CSV file!\n");
tft.fillScreen(background_color);
tft.drawString("Wio Terminal", 35, 105);
tft.drawString("cannot open the file!", 35, 125);
}
// Exit and clear:
delay(3000);
tft.fillScreen(background_color);
drawImage<uint16_t>("carpal_tunnel.bmp", TFT_HEIGHT/2, 0);
drawImage<uint16_t>("mouse.bmp", TFT_HEIGHT/2, TFT_WIDTH-90);
}⭐ According to the pressed configurable button (Button A, Button B, or Button C), append the data record, including the selected muscle soreness class, to the given CSV file.
if(digitalRead(WIO_KEY_A) == LOW) save_data_to_SD_Card("0");
if(digitalRead(WIO_KEY_B) == LOW) save_data_to_SD_Card("1");
if(digitalRead(WIO_KEY_C) == LOW) save_data_to_SD_Card("2");After uploading and running the code for collecting forearm muscle soreness data and saving information to a given CSV file on the SD card on Wio Terminal:
💪💻 The device displays real-time GSR and EMG measurements as line charts on the TFT screen.
💪💻 If Button A (configurable button) is pressed, the device appends a new row (data record) to the mouse_fatigue_data_set.csv file on the SD card by adding the Relaxed [0] muscle soreness class under the Soreness data field. Then, if the device saves the data record successfully to the given CSV file on the SD card, it shows:
💪💻 If Button B (configurable button) is pressed, the device appends a new row (data record) to the mouse_fatigue_data_set.csv file on the SD card by adding the Tense [1] muscle soreness class under the Soreness data field. Then, if the device saves the data record successfully to the given CSV file on the SD card, it shows:
💪💻 If Button C (configurable button) is pressed, the device appends a new row (data record) to the mouse_fatigue_data_set.csv file on the SD card by adding the Exhausted [2] muscle soreness class under the Soreness data field. Then, if the device saves the data record successfully to the given CSV file on the SD card, it shows:
💪💻 If Wio Terminal cannot open the given CSV file successfully, the device displays the error message on the TFT screen.
💪💻 Also, the device prints notifications and sensor measurements on the serial monitor for debugging.
After logging the collected forearm muscle soreness data in a CSV file on the SD card for 15 days, I elicited my data set with eminent validity :)
When I completed collating my forearm muscle soreness data set and assigning labels, I had started to work on my artificial neural network model (ANN) to make predictions on muscle soreness levels (classes) based on GSR and EMG measurements.
I decided to create my neural network model with TensorFlow in Python. Thus, first of all, I followed the steps below to grasp a better understanding of my data set so as to train my model accurately:
- Data Visualization
- Data Scaling (Normalizing)
- Data Preprocessing
- Data Splitting
As explained in the previous steps, I assigned muscle soreness classes empirically while logging data with the device. Since the assigned classes are stored under the Soreness data field in the mouse_fatigue_data_set.csv file, I preprocessed my data set effortlessly to assign labels for each data record (input):
- 0 — Relaxed
- 1 — Tense
- 2 — Exhausted
After scaling (normalizing) and preprocessing inputs, I obtained two input variables and one label for each data record in my data set. Then, I built an artificial neural network model with TensorFlow and trained it with my training data set to acquire the best results and predictions possible.
Layers:
- 2 [Input]
- 64 [Hidden]
- 32 [Hidden]
- 8 [Hidden]
- 3 [Output]
To execute all steps above and convert my model from a TensorFlow Keras H5 model to a C array (.h file) so as to run it successfully on Wio Terminal, I developed an application in Python. As shown below, the application consists of three code files and three folders:
- main.py
- test_data.py
- tflite_to_c_array.py
- /data
- /model
- /bmp
First of all, I created a class named Mouse_Fatigue in the main.py file to execute the following functions precisely.
⭐ Include the required modules.
import tensorflow as tf
from tensorflow import keras
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from tflite_to_c_array import hex_to_c_array
from test_data import test_inputs, test_labels⭐ In the __init__ function, define the required variables to build the neural network model and read the forearm muscle soreness data set from the given CSV file.
def __init__(self, csv_path):
self.inputs = []
self.labels = []
self.model_name = "mouse_fatigue_level"
self.scale_val = 1000
# Read the collated soreness data set (GSR and EMG):
self.df = pd.read_csv(csv_path)I will elucidate each file and function in detail in the following steps.
Note: The bmp folder is thoroughly explained in Step 2.1.
Since it is essential to understand a given data set to pass appropriately formatted inputs and labels to a neural network model, I decided to visualize my data set and scale (normalize) it in Python after extracting it from the mouse_fatigue_data_set.csv file saved in the data folder.
⭐ In the graphics function, visualize the requested data column (field) from the given data set by utilizing the Matplotlib library.
def graphics(self, column_1, column_2, x_label, y_label):
# Show the requested data column from the data set:
plt.style.use("dark_background")
plt.gcf().canvas.set_window_title('Mouse Fatigue Estimation by GSR and EMG Values')
plt.hist2d(self.df[column_1], self.df[column_2], cmap="coolwarm")
plt.colorbar()
plt.xlabel(x_label)
plt.ylabel(y_label)
plt.title(x_label)
plt.show()⭐ In the data_visualization function, scrutinize all data columns (fields) before scaling and preprocessing the given data set so as to build a neural network model with appropriately formatted data.
def data_visualization(self):
# Scrutinize data columns to build a model with appropriately formatted data:
self.graphics('GSR', 'EMG', 'GSR', 'EMG')After visualizing my data set, I needed to create inputs from data records to train my neural network model. Therefore, I utilized these two data elements to create inputs:
- GSR
- EMG
Then, I scaled (normalized) each data element to format them properly and thus extracted these scaled data elements from my data set for each data record:
- scaled_GSR
- scaled_EMG
⭐ In the scale_data_and_define_inputs function, divide every data element into their required values so as to make them smaller than or equal to 1.
⭐ Then, create inputs with the scaled data elements, append them to the inputs array, and convert this array to a NumPy array by using the asarray function.
⭐ Each input includes two parameters [shape=(2, )]:
- [0.152, 0.407]
def scale_data_and_define_inputs(self):
self.df["scaled_GSR"] = self.df["GSR"] / self.scale_val
self.df["scaled_EMG"] = self.df["EMG"] / self.scale_val
# Create the inputs array by utilizing the scaled variables:
for i in range(len(self.df)):
self.inputs.append(np.array([self.df["scaled_GSR"][i], self.df["scaled_EMG"][i]]))
self.inputs = np.asarray(self.inputs)As explained in the previous steps, I assigned muscle soreness classes for each data record under the Soreness data field to be utilized as labels:
- 0 — Relaxed
- 1 — Tense
- 2 — Exhausted
⭐ In the define_and_assign_labels function, get predefined labels [0 - 2] under the Soreness data field for each input and append them to the labels array.
def define_and_assign_labels(self):
self.labels = self.df["Soreness"]Since my forearm muscle soreness data set is already limited, I decided to utilize all of my data set as the training data set instead of splitting it into training and testing data sets. Thus, I created a separate testing data set in the test_data.py file.
After defining the training data set, I scaled (normalized) the testing data set inputs to format them appropriately.
def split_data(self):
# (training)
self.train_inputs = self.inputs
self.train_labels = self.labels
# (test)
self.test_inputs = test_inputs / self.scale_val
self.test_labels = test_labelsThen, I built my artificial neural network model (ANN) by utilizing Keras and trained it with the training set for 150 epochs.
You can inspect these tutorials to learn about activation functions, loss functions, epochs, etc.
def build_and_train_model(self):
# Build the neural network:
self.model = keras.Sequential([
keras.Input(shape=(2,)),
keras.layers.Dense(64, activation='relu'),
keras.layers.Dense(32, activation='relu'),
keras.layers.Dense(8, activation='relu'),
keras.layers.Dense(3, activation='softmax')
])
# Compile:
self.model.compile(optimizer='adam', loss="sparse_categorical_crossentropy", metrics=['accuracy'])
# Train:
self.model.fit(self.train_inputs, self.train_labels, epochs=150)
...
After training with the training set (inputs and labels), the accuracy of my neural network model is between 0.83 and 0.88.
After building and training my artificial neural network model, I tested its accuracy and validity by utilizing the testing data set (inputs and labels).
The evaluated accuracy of the model is 0.9375.
...
# Test the model accuracy:
print("\n\nModel Evaluation:")
test_loss, test_acc = self.model.evaluate(self.test_inputs, self.test_labels)
print("Evaluated Accuracy: ", test_acc)
After evaluating my neural network model, I saved it as a TensorFlow Keras H5 model (mouse_fatigue_level.h5) to the model folder.
def save_model(self):
self.model.save("model/{}.h5".format(self.model_name))However, running a TensorFlow Keras H5 model on Wio Terminal to make predictions on muscle soreness levels is not eligible and efficient considering size, latency, and power consumption.
Thus, I converted my neural network model from a TensorFlow Keras H5 model (.h5) to a TensorFlow Lite model (.tflite). Then, I modified the TensorFlow Lite model to create a C array (.h file) to run the model on Wio Terminal successfully.
To revise the TensorFlow Lite model as a C array, I applied the hex_to_c_array function copied directly from this tutorial to the tflite_to_c_array.py file.
⭐ In the convert_TF_model function, convert the recently trained and evaluated model to a TensorFlow Lite model by applying the TensorFlow Lite converter (tf.lite.TFLiteConverter.from_keras_model).
⭐ Then, save the generated TensorFlow Lite model to the model folder (mouse_fatigue_level.tflite).
⭐ Modify the saved TensorFlow Lite model to a C array (.h file) by executing the hex_to_c_array function.
⭐ Finally, save the generated C array to the model folder (mouse_fatigue_level.h).
def convert_TF_model(self, path):
#model = tf.keras.models.load_model(path + ".h5")
converter = tf.lite.TFLiteConverter.from_keras_model(self.model)
#converter.optimizations = [tf.lite.Optimize.DEFAULT]
#converter.target_spec.supported_types = [tf.float16]
tflite_model = converter.convert()
# Save the recently converted TensorFlow Lite model.
with open(path + '.tflite', 'wb') as f:
f.write(tflite_model)
print("\r\nTensorFlow Keras H5 model converted to a TensorFlow Lite model!\r\n")
# Convert the recently created TensorFlow Lite model to hex bytes (C array) to generate a .h file string.
with open("model/{}.h".format(self.model_name), 'w') as file:
file.write(hex_to_c_array(tflite_model, self.model_name))
print("\r\nTensorFlow Lite model converted to a C header (.h) file!\r\n")After building, training, and converting my neural network model to a C array (.h file), I needed to upload and run my model directly on Wio Terminal so as to create an easy-to-use and capable device without any dependencies.
Plausibly, TensorFlow provides an official library to run inferences on microcontrollers with the TensorFlow Lite models converted to C arrays. Although, for now, it only supports a few development boards, including Wio Terminal, it is a fast and efficient library for running basic neural network models on microcontrollers. Since TensorFlow Lite for Microcontrollers does not require operating system support, any standard C or C++ libraries, or dynamic memory allocation, Wio Terminal can forecast forearm muscle soreness levels without needing Wi-Fi connectivity.
#️⃣ To download the official TensorFlow library on the Arduino IDE, go to Sketch ➡ Include Library ➡ Manage Libraries… and search for TensorFlow. Then, install the latest version of the Arduino_TensorFlowLite library.
After installing the TensorFlow library on the Arduino IDE, I needed to import my neural network model modified as a C array (mouse_fatigue_level.h) to run inferences.
#️⃣ To import the TensorFlow Lite model converted as a C array (.h file), go to Sketch ➡ Add File...
After importing the converted model (.h file) successfully to the Arduino IDE, I modified the code in this tutorial from TensorFlow to run my neural network model. Then, I employed the 5-way switch integrated into Wio Terminal to run inferences so as to forecast forearm muscle soreness levels:
- Press ➡ Run Inference
You can download the mouse_fatigue_detection_run_model.ino file to try and inspect the code for running TensorFlow Lite neural network models on Wio Terminal.
You can inspect the corresponding functions and settings in Step 3.
⭐ Include the required libraries.
#include <SPI.h>
#include <Seeed_FS.h>
#include "TFT_eSPI.h"
#include "seeed_line_chart.h"
#include "SD/Seeed_SD.h"
#include "RawImage.h"⭐ Import the required TensorFlow modules.
#include "TensorFlowLite.h"
#include "tensorflow/lite/micro/kernels/micro_ops.h"
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/all_ops_resolver.h"
#include "tensorflow/lite/version.h"⭐ Include the TensorFlow Lite model modified as a C array (.h file).
⭐ Define the TFLite globals used for compatibility with Arduino-style sketches.
⭐ Create an area of memory to utilize for input, output, and other TensorFlow arrays.
#include "mouse_fatigue_level.h"
// TFLite globals, used for compatibility with Arduino-style sketches:
namespace {
tflite::ErrorReporter* error_reporter = nullptr;
const tflite::Model* model = nullptr;
tflite::MicroInterpreter* interpreter = nullptr;
TfLiteTensor* model_input = nullptr;
TfLiteTensor* model_output = nullptr;
// Create an area of memory to use for input, output, and other TensorFlow arrays.
constexpr int kTensorArenaSize = 15 * 1024;
uint8_t tensor_arena[kTensorArenaSize];
} // namespace⭐ Define the threshold value (0.75) for the model outputs (results).
⭐ Define the muscle soreness level (class) names and color codes:
- Relaxed
- Tense
- Exhausted
float threshold = 0.75;
// Define the muscle soreness level (class) names and color codes:
String classes[] = {"Relaxed", "Tense", "Exhausted"};
uint32_t color_codes[] = {tft.color565(1,156,0), tft.color565(255,169,2), tft.color565(226,16,1)};⭐ Then, define the image (16-bit BMP file) list to be displayed on the TFT screen for each muscle soreness level after being detected.
const char* images[] = {"relaxed.bmp", "tense.bmp", "exhausted.bmp"};⭐ Define the TensorFlow Lite model settings:
⭐ Set up logging (will report to Serial, even within TFLite functions).
⭐ Map the model into a usable data structure.
static tflite::MicroErrorReporter micro_error_reporter;
error_reporter = µ_error_reporter;
// Map the model into a usable data structure.
model = tflite::GetModel(mouse_fatigue_level);
if (model->version() != TFLITE_SCHEMA_VERSION) {
error_reporter->Report("Model version does not match Schema");
while(1);
}⭐ Pull all the operation implementations.
⭐ Build an interpreter to run the model.
// This pulls in all the operation implementations we need.
// NOLINTNEXTLINE(runtime-global-variables)
static tflite::AllOpsResolver resolver;
// Build an interpreter to run the model.
static tflite::MicroInterpreter static_interpreter(
model, resolver, tensor_arena, kTensorArenaSize,
error_reporter);
interpreter = &static_interpreter;⭐ Allocate memory from the tensor_arena for the model's tensors.
⭐ Assign model input and output buffers (tensors) to pointers.
TfLiteStatus allocate_status = interpreter->AllocateTensors();
if (allocate_status != kTfLiteOk) {
error_reporter->Report("AllocateTensors() failed");
while(1);
}
// Assign model input and output buffers (tensors) to pointers.
model_input = interpreter->input(0);
model_output = interpreter->output(0);⭐ Define and display the required 16-bit images saved on the SD card.
for(int i=0; i<3; i++){ drawImage<uint16_t>(images[i], TFT_HEIGHT, 0); }
drawImage<uint16_t>("carpal_tunnel.bmp", TFT_HEIGHT/2, 0);
drawImage<uint16_t>("mouse.bmp", TFT_HEIGHT/2, TFT_WIDTH-90);⭐ In the run_inference_to_make_predictions function:
⭐ Scale (normalize) the collected forearm muscle soreness data depending on the given model and copy them to the input buffer (tensor).
⭐ Run inference.
⭐ Read predicted y values (muscle soreness classes) from the output buffer (tensor).
⭐ Obtain the detection result greater than the given threshold (0.75). It represents the most accurate label (muscle soreness class) predicted by the model.
⭐ Then, display the detected muscle soreness class with its assigned image and color code on the TFT screen.
void run_inference_to_make_predictions(){
// Scale (normalize) muscle soreness data depending on the given model and copy them to the input buffer (tensor):
model_input->data.f[0] = gsr_value / 1000;
model_input->data.f[1] = emg_value / 1000;
// Run inference:
TfLiteStatus invoke_status = interpreter->Invoke();
if (invoke_status != kTfLiteOk) {
error_reporter->Report("Invoke failed on the given input.");
}
// Read predicted y values (muscle soreness classes) from the output buffer (tensor):
for(int i = 0; i<3; i++){
if(model_output->data.f[i] >= threshold){
int w = 150, h = 40, str_x = 12, str_y = 65;
int y_offset = h + ((h - str_y) / 2);
int x_offset = classes[i].length() * str_x;
// Display the detection result (class).
tft.fillScreen(background_color);
drawImage<uint16_t>(images[i], (TFT_HEIGHT-75)/2, 0);
// Print:
tft.fillRect((TFT_HEIGHT-w)/2, TFT_WIDTH-h, w, h, color_codes[i]);
tft.drawString(classes[i], (TFT_HEIGHT-x_offset)/2, TFT_WIDTH-y_offset);
}
}
// Exit and clear:
delay(3000);
tft.fillScreen(background_color);
drawImage<uint16_t>("carpal_tunnel.bmp", TFT_HEIGHT/2, 0);
drawImage<uint16_t>("mouse.bmp", TFT_HEIGHT/2, TFT_WIDTH-90);
}⭐ If the 5-way switch is pressed, run inference to predict forearm muscle soreness level (class).
if(digitalRead(WIO_5S_PRESS) == LOW) run_inference_to_make_predictions();My neural network model predicts possibilities of labels (muscle soreness classes) for each given input as an array of 3 numbers. They represent the model's "confidence" that the given input array corresponds to each of the three different muscle soreness classes based on GSR and EMG measurements [0 - 2], as shown in Step 4:
- 0 — Relaxed
- 1 — Tense
- 2 — Exhausted
After importing and setting up my neural network model as a C array on Wio Terminal successfully, I utilized the model to run inferences to forecast muscle soreness levels.
After executing the code for running inferences on Wio Terminal:
💪💻 The device displays real-time GSR and EMG measurements as line charts on the TFT screen.
💪💻 If the 5-way switch is pressed, the device runs inference with the model by employing the most recently generated GSR and EMG measurements as the input.
💪💻 Then, the device displays the output, which represents the most accurate label (muscle soreness class) predicted by the model.
💪💻 Each muscle soreness level (class) has a unique image (16-bit BMP file) and color code to be shown on the TFT screen when being detected as the output:
As far as my experiments go, the device operates impeccably while predicting forearm muscle soreness levels (classes) :)
After completing all steps above and experimenting, I have employed the device to predict and detect forearm muscle soreness levels so as to get prescient warnings to prevent permanent or temporary disorder risks related to mouse overuse such as Repetitive strain injury (RSI) and Carpal tunnel syndrome (CTS).
By applying neural network models trained on GSR and EMG measurements in detecting muscle soreness levels while using the mouse, we can achieve to:
💪💻 alleviate the exacerbating effects of mouse overuse,
💪💻 palliate temporary symptoms of overexposure to monotonous mouse movements,
💪💻 prevent permanent or temporary disorder risks related to mouse overuse such as Repetitive strain injury (RSI) and Carpal tunnel syndrome (CTS),
💪💻 avert permanent mouse-related injuries.
[1] Computer Mouse - Common Problems from Use, Canadian Centre for Occupational Health & Safety, June 2017, https://www.ccohs.ca/oshanswers/ergonomics/office/mouse/mouse_problems.html.
[2] Ergonomics & Computer Use, The Trustees of Princeton University, https://uhs.princeton.edu/health-resources/ergonomics-computer-use.
Comments