Posture Watchdog

Overview

Before COVID19 people had to do some physical activity willingly or unwillingly in their normal life activities. For example they had to commute to schools, offices or business places. During the COVID19 many people have changed their lifestyle. Most of them have started working remotely from home with restricted physical activities. Also, in schools or offices you have to maintain your sitting postures but at home you are free to seat or lay in any posture at your own comfort or out of laziness while working. The lack of activities and bad postures are posing severe health issues in the long run. The major problem is the spinal pain caused by the poor sitting posture on the office chair. In this project I have built a proof of concept for an activity and posture monitoring and alert wearable device which tells the user their activities or idleness and alert in time to make them active or ask to correct the posture. I will try my best to provide step by step instructions if anybody wants to recreate the project.

Setup Development Environment

I am using a MacOS for the development. First we need to download nRF connect for Desktop from here: https://www.nordicsemi.com/Software-and-tools/Development-Tools/nRF-Connect-for-desktop/Download. The nRF Connect for Desktop is a cross-platform tool that enables testing and development with nRF5340. Please follow the installation guide in the link above. After installation open the app and click on the Toolchain Manager to install nRF Connect SDK.

By default it is installed in the /opt/nordic/ncs in MacOS. Since we are going to use Adafruit display shield with nRF5340 DK, the current version of the nrf5340 board files in Zephyr does not fully support the Arduino definitions required by the Adafruit display shield. In order to fix this we need to apply a patch. Please follow commands below to apply the patch.

$ cd /tmp
$ git clone https://github.com/NordicPlayground/ncs-display-ble-example.git
$ cd ncs-display-ble-example
$ cp nrf5340_display.patch /opt/nordic/ncs/v1.5.1/zephyr
$ cd /opt/nordic/ncs/v1.5.1/zephyr
$ git apply nrf5340_display.patch

Also, we are going to use inbuilt microSD card in the Adafruit shield with the nRF5340 DK. We need to change arduino_header pin definition from 12 to 11 in the boards/shields/adafruit_2_8_tft_touch_v2/adafruit_2_8_tft_touch_v2.overlay file as shown below.

<&arduino_header 11 GPIO_ACTIVE_LOW>; /* D04 */

Once the environment setup is completed we have to activate it by executing the command below.

$ source /opt/nordic/ncs/v1.5.1/zephyr/zephyr-env.sh

Hardware Setup

The nRF5340 DK with display shield is used as a BLE central device. The nRF52840 dongle connected to the ADXL345 accelerometer is used as a BLE peripheral. The connection diagram schematics can be found in the Schematic section. The physical connection is shown in the image below.

We need to add an overlay file for the ADXL345 accelerometer in the BLE peripheral application which defines accelerometer I2C pin definitions and the slave address.

&i2c1 {
  compatible = "nordic,nrf-twim";
  status = "okay";
  sda-pin = < 31 >;
  scl-pin = < 29 >;
  clock-frequency = <I2C_BITRATE_STANDARD>;
  adxl345@53 {
    compatible = "adi,adxl345";
    reg = < 0x53 >;
    label = "ADXL345";
 };
};

Application

For this proof of concept 3 projects are created.

• nrf52840_dongle_ble_peripheral_acc (as a data node)
• nrf5340_dk_ble_central_acc (for data collection)
• nrf5340_dk_ble_central_inf (for model inferencing)

All projects' code can be found at the Github repository: https://github.com/metanav/AdvancedWearables

Data Collection

The training data is collected for 4 classes:

• Running
• Walking
• Sitting Good
• Sitting Bad

The BLE peripheral (nRF52840 dongle) collects the 3-axis accelerometer data at 25Hz sample rate. Every second it notifies the connected central device with 150 bytes [25 (Hz) x 3 (axis) x 2 (16 bit)] payload. An easy to use graphical user interface (GUI) is designed using the display shield which allows to select the training label and stores the data. The BLE central device collects the accelerometer data from the peripheral and saves it into the SD card (inbuilt in the display shield). The 3-axis accelerometer data is saved into CSV file format with filename as the label concatenated with an auto increment integer. A sample is shown below:

Filename: Sitting_Bad.4.csv

-187,-937,218
-187,-937,218
-187,-937,218
-187,-906,218
-187,-906,218
-218,-906,218
-218,-906,218
-187,-937,218
-218,-906,218
-218,-906,218
-187,-906,218
-187,-906,218
-187,-906,218
-218,-906,218
-187,-906,218
-218,-906,218
-218,-906,218
-218,-906,218
-218,-906,218
-218,-906,218
-187,-906,218

Build and flash the data collection firmware for nRF5340 DK

Connect the nRF5340 DK to the computer using USB cable and run the commands:

$ pip3 install -U west
$ git clone https://github.com/metanav/AdvancedWearables
$ cd AdvancedWearables/nrf5340_dk_ble_central_acc
$ source /opt/nordic/ncs/v1.5.1/zephyr/zephyr-env.sh
$ rm -rf build
$ west build -b nrf5340dk_nrf5340_cpuapp
$ west flash

Build and flash firmware for nRF52840 Dongle

Connect the nRF52840 dongle to the computer and run the commands:

$ cd AdvancedWearables/nrf52840_dongle_ble_peripheral_acc
$ rm -rf build
$ west build -b nrf52840dongle_nrf52840

We cannot use west to flash nRF52840. We will use nRF Connect > Programmer. First press the Reset button at the dongle and select the device from the top left dropdown menu. Click at Add Hex file button > browse... and select AdvancedWearables/nrf52840_dongle_ble_peripheral_acc/build/zephyr/zephyr.hex file and click at Write button.

Data Collection Demo

Data collection demo (real time)

Data upload to Edge Impulse Studio

The firmware created by Edge Impulse Studio for the nRF5340 DK is convenient and it uploads data automatically coming from USB UART connection to the DK. But I wanted to make a wearable device which can be used for data collection as well as model inferencing in the field. It is inconvenient to carry laptop while data collection when you are walking or running. Also, you may not have internet connectivity in the field. So I created completely standalone device which captures the data from the peripheral device over BLE and saves in to microSD card. The captured data is copied to the computer. I have written a script (below) which converts the raw accelerometer data into the data acquisition JSON format required by the Edge Impulse studio for training.

import json
import time
import hmac
import hashlib
import os
import configparser

config = configparser.ConfigParser()
config.read('config.ini')
HMAC_KEY = config['edge_impulse']['hmac_key']

dir = 'raw_data'

for filename in os.listdir(dir):
    if filename.endswith('.csv'):
        prefix, ext = os.path.splitext(filename)
        outfilename = os.path.join('formatted_data', '{}.json'
                             .format(prefix))
        values = []
        with open(os.path.join(dir, filename)) as fp:
            for line in fp:
                line = line.strip()
                values.append([int(i) for i in line.split(',')])
                emptySignature = ''.join(['0'] * 64)
                data = {
                    "protected": {
                        "ver": "v1",
                        "alg": "HS256",
                        "iat": time.time()
                    },
                    "signature": emptySignature,
                    "payload": {
                        "device_name": "E1:01:2D:53:40:DA",
                        "device_type": "NRF5340DK",
                        "interval_ms": 40,
                        "sensors": [
                            { "name": "accX", "units": "m/s2" },
                            { "name": "accY", "units": "m/s2" },
                            { "name": "accZ", "units": "m/s2" }
                        ],
                        "values": values
                    }
                }   
                # encode in JSON
                encoded = json.dumps(data)
                # sign message
                signature = hmac.new(bytes(HMAC_KEY, 'utf-8'), 
                                msg = encoded.encode('utf-8'), 
                                digestmod = hashlib.sha256).hexdigest()
                # set the signature again in the message, and encode again
                data['signature'] = signature
                encoded = json.dumps(data, indent=4)
                with open(outfilename, 'w') as fout:
                    fout.write(encoded)

First copy all raw data from the microSD card to the computer into a directory raw_data. You need to register an account at Edge Impulse and create a new project to upload data.

Also, you need an HMAC key for your Edge Impulse Studio project to generate signature for the data acquisition format. You can copy the HMAC key from the Dashboard > Keys [tab] as shown below.

Create a config.ini file with the contents below with your own HMAC key.

[edge_impulse]
hmac_key = <your hmac key>

Execute the following command.

$ mkdir formatted_data
$ python3 data_aquisition_format.py

The converted data looks like as shown below. The sample rate is 25 Hz so the interval_ms is 40ms which can be changed in the script if your sensor sample rate is different.

{
  "protected": {
    "ver": "v1",
    "alg": "HS256",
    "iat": 1623920716.465548
  },
  "signature": "19bec321f71e504460add5f453fc12d636153f06ebac2f3f6ba4a49813",
  "payload": {
  "device_name": "E1:01:2D:53:40:DA",
  "device_type": "NRF5340DK",
  "interval_ms": 40,
  "sensors": [
     { "name": "accX", "units": "m/s2" },
     { "name": "accY","units": "m/s2" },
     { "name": "accZ", "units": "m/s2" }
   ],
   "values": [
     [ -187, -906, 250 ],
     [ -218, -906, 250 ],
     [ -187, -906, 250 ],
     [ -187, -875, 250 ],
     // more values
    ]
  }
}

The data is uploaded using the Edge Impulse CLI. Please follow the instructions to install the CLI here: https://docs.edgeimpulse.com/docs/cli-installation. The command below is used to upload all JSON files which is automatically split into training and testing datasets.

$ cd formatted_data
$ edge-impulse-uploader --category split *.json

We can see the uploaded data in the Data Acquisition page.

Training

Go to the Impulse Design > Create Impulse page and click at the Add a processing block and choose Spectral Analysis which is great for analyzing repetitive data from accelerometers. It extracts the frequency and power characteristics of a signal over time. Also, at the same page click at the Add a learning block and choose Neural Network(Keras) which learns patterns from data, and can apply these to new data. it is a good fit for categorizing movement data. Since movement recognition for Sitting state needs a bit longer duration therefore for the Time series data block, I have chosen 2000ms Window size and 100ms Window increase. Now click on the Save Impulse button.

Now go to the Impulse Design > Spectral Features page and click at Save parameters button. I have kept the default values which looks good to me.

Clicking on Save parameters button redirects to another page where we should click on Generate Feature button. It usually takes couple of minutes to complete feature generation. We can see the 3D visualization of the generated features in the Feature Explorer. It seems data has clear partition for Walking and Running but Sitting postures are somewhat near to each other. It would be Neural Network job to train and learn the patterns and make better separation among the classes.

Now go to the Impulse Design > NN Classifier page and define Neural Network architecture. I have created the model with 4 dense (fully connected) layers which are great for the output of a spectral analysis DSP block. The number of training cycles was chosen as 50.

Now click at Start Training button and wait few minutes until training is completed. We can see the Training output below. The model has 86.8% accuracy which is not bad for such a small size dataset and we still have room for improvement. Also, the Confusion matrix clearly shows the Sitting_Good and Sitting_Bad classes have false positives which can be improved by collecting more data for these two classes. For this proof of concept project we will use this model.

Deployment

The Edge Impulse Studio supports the nRF5340 DK deployment firmware binaries but for this project we are using other functionalities for example connecting to the peripheral device to get data and display results on the Adafruit display shield. So in the Deployment page we will choose Create Library > C++ Library option. For the Select optimization option, we will choose Enable EON Compiler which reduces the memory usage of the model. Also, we will opt for Unoptimized (Float32) model since it had better accuracy than the Quantized (Int8) model. Now click at the Build button and in few seconds the library bundle will be downloaded at the local computer.

For inferencing I have created a new nRF5340 DK project which borrows most of the BLE and display code from the data collection project. Since inferencing library has mostly C++ codebase we need to change few things in the project settings.

• Rename all *.c files to *.cpp in the src directory copied from the data collection project
• Use extern "C" in few header files
• Add C++ config in the prj.conf

CONFIG_CPLUSPLUS=y
CONFIG_LIB_CPLUSPLUS=y
CONFIG_NEWLIB_LIBC=y
CONFIG_NEWLIB_LIBC_FLOAT_PRINTF=y
CONFIG_FPU=y

Now unzip and move following files to the project root directory.

• CMakeLists.txt
• edge-impulse-sdk
• model-parameters
• tflite-model

Build and flash the inferencing firmware for nRF5340 DK

Connect the nRF5340 DK to the computer using USB cable and run the following commands:

$ git clone https://github.com/metanav/AdvancedWearables 
$ cd AdvancedWearables/nrf5340_dk_ble_central_inf
$ source /opt/nordic/ncs/v1.5.1/zephyr/zephyr-env.sh
$ rm -rf build
$ west build -b nrf5340dk_nrf5340_cpuapp
$ west flash

Inferencing Demo

The BLE Central (nRF5340) collects 2000ms of the accelerometer data from the BLE peripheral and run inferencing. The inferencing log (below) shows the DSP time is 8ms and Classification time is 1ms which is blazingly fast. Since inferencing speed is fast so it does not add up to the total lagging of 2 seconds (for data accumulation from the peripheral) for each inferencing cycle.

Predictions (DSP: 8 ms., Classification: 1 ms., Anomaly: 0 ms.)

Features (8 ms.): 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 2.509764 0.793651 1.043532 3.571429 0.199363 0.000000 0.000000 0.128670 0.626574 0.175257 0.013350 

Running neural network...
Predictions (time: 1 ms.):
Running:.0.000000
Sitting_Bad:.0.226727
Sitting_Good:.0.773085
Walking:.0.000188

Conclusion

This project was started as an application to test out nRF5340 DK abilities for machine learning wearable project. At starting it seemed daunting task but after spending some time and learning the SDK it was easy and satisfying. It turned out that nRF5340 DK is a capable piece of hardware which is a good fit for TinyML. Also, the Edge Impulse Studio is an easy to use online tool to train and deploy machine learning models for quick prototyping. I would like to thank Nordic Semiconductor to provide me free hardware for this contest and forcing me to brush up my rusty C/C++ skills.