Drowning Detection Device using Machine Learning

In Vietnam, drowning is the second leading cause of death in children aged 5-14, with an average of 6-7 children under 15 dying every day (Children's Protection Hotline 111). Thus, drowning is a severe problem and needs everyone's attention. To detect early and prevent drowning, I propose a wearable device that integrates machine learning technology, signal processing algorithms, remote wireless communication technology, and an accelerometer with a pressure sensor on a compact microcontroller. I believe that my product will help prevent drowning not only for children but also for anyone who uses the product.

Theory of Operation

The control device uses an accelerometer worn on the user's back to collect movement information, as shown below:

(Link to the research paper: https://www.researchgate.net/publication/324508656_Wearable_sensors_and_smart_equipment_for_feedback_in_watersports )

With this data, we can see that there are similarities in the data patterns. When analyzing this data in terms of period and frequency, we can extract information about movement intensity, and arm stroke amplitude, thereby accurately identifying the person’s current movement underwater.

Combine this data with biomedical knowledge about human actions when there are signs that lead to drowning such as struggling, the arms stiffly stretched out to the sides, the body facing up but the legs unable to move because the lung is flooded,... we can know exactly the stage of drowning that the person is in.

However, an expert is still needed to analyze this data. To solve this problem, I applied amachine learning service from Edge Impulse with the ability to recognize data patterns with an accuracy of more than 85.7% to extract information from data patterns.

Pressure sensors will be deployed to measure the depth of the user and use the change in pressure to calculate the vertical movement velocity.

The pressure of water column is calculated with this formula:

Water Pressure = Density of Water x gForce Acceleration x Depth
               = 1000 kg/m^3 x 9.81 m/s^2 x 1 m = 9.81 kPa

But since there is not only water but also atmospheric pressure, air pressure needs to be added to measure the total pressure.

At 1 m depth underwater the pressure is:

P.air + P.water = 101.325 kPa + 9.81 kPa = 111.135 kPa

Hardware Build

1. Pressure Sensor

2. Accelerometer

3. XIAO ESP32C3

4. Power switch

5. 300mAh battery

6. Antenna

Data Collection

First, we have to collect data for machine learning model training. To do so, we flash a program to read the acceleration sensor reading, then connect it to the Edge Impulse server to store and train the model.

Thanks to mjrobot, we will flash the following sensor reading program into our Seeed XIAO ESP32C3:

The Platform.ini of the project should look like this:

[env:seeed_xiao_esp32c3]
platform = espressif32
board = seeed_xiao_esp32c3
framework = arduino
build_flags = -D PIO_FRAMEWORK_ARDUINO_MMU_CACHE16_IRAM48_SECHEAP_SHARED
monitor_speed = 115200
lib_deps= https://github.com/jrowberg/i2cdevlib.git
https://github.com/ElectronicCats/mpu6050.git

Calibration can be performed for more accurate data before outputting data to the server. To do this, the user just has to run the following sketch, open the serial monitor, and wait for the offset value to appear after some time.

I use the same project configuration for this calibration program as the above program so the Platform.ini file should look the same as above

Then insert those values into the Set.AccelOfset is part of the original data-collecting program. Note that the data format should look like the following:

For further understanding of the data collection code, you can refer to this amazing post from mjrobot

After the device is loaded with the correct program, we will start connecting the device to the Edge Impulse service.

First, create an account on Edge Impulse by following this link: https://edgeimpulse.com/, then follow this link to install Node.js and the Edge Impulse CLI, this step is crucial as we will use them after

On the Edge Impulse website, create your project:

and now connect the device to the server via the command prompt. Make sure to connect the Seeed XIAO ESP32C3 to your computer first before doing the next step. Open the command prompt, and input this command:

edge-impulse-data-forwarder --clean

After some time, the CLI will ask you to insert your user credentials and the names of all your data.

Model Training

I first captured data when simulating a drowning experience, this is 2 examples when collecting data, you must simulate the drowning experience yourself.

Below is one sample of 10 seconds:

I captured 3m 50s (23 samples of 10 seconds each)

Below is our final Edge Impulse design:

Generating Features:

First, let’s take the raw data and convert it to tabular data. Go to the spectral Features tab and select Save Parameters:

After that, I use a classifier to classify data to detect different types of movement:

For anomaly detection, I used these features. The cluster count will be 32, as suggested by the Studio

Implementation

The ML model is on the Edge Impulse website so you have to implement it on our XIAO ESP32C3 board. To do so, first go to the deployment page then select the option Arduino Library and Build. A Zip file will be created and downloaded to your computer.

Then unzip the file. Don’t forget to remember the destination path of the file.

Create a Platform project, then put the library path in Platform.ini. Your Platform.ini should look like this:

[env:seeed_xiao_esp32c3]
platform = espressif32
board = seeed_xiao_esp32c3
framework = arduino
build_flags = -D PIO_FRAMEWORK_ARDUINO_MMU_CACHE16_IRAM48_SECHEAP_SHARED
monitor_speed = 115200
board_build.partitions = no_ota.csv
lib_deps= 
C:\users\admin\Documents\Library_Machine_Learning\ei-minh_quang2304-project-1-arduino-1.0.4\minh_quang2304-project-1_inferencing
          https://github.com/jrowberg/i2cdevlib.git
          https://github.com/ElectronicCats/mpu6050.git
          jrowberg/I2Cdevlib-BMP085 @ ^1.0.0

Since the actual program size is very large, you may want to use the no_ota build scheme. To do that refer to this link to download the no_ota.csv file. Your project folder should look something like this:

Adjustments

Since XIAO ESP32C3 is not directly supported by the Edge Impulse service, you will have to follow through a series of steps to configure the library to be compatible with the XIAO ESP32C3.

First, open the example folder of your model library:

In the nano_ble33_sense folder, select nano_ble33_sense_accelerometer. And in this folder, you will see an INO file. Click on it and open it with Notepad then copy the whole sketch:

And paste it into your PlatformIO project. Now you have to make some changes to the program:

#include <minh_quang2304-project-1_inferencing.h>
#include "I2Cdev.h"
#include "MPU6050.h"
#include "Wire.h"
#include "BMP085.h"
#include <Arduino.h>

Then change some constant define:

// Constant Define
#define ACC_RANGE 1 // 0: -/+2G; 1: +/-4G
#define CONVERT_G_TO_MS2 (9.81 / (16384 / (1. + ACC_RANGE)))
#define MAX_ACCEPTED_RANGE (2 * 9.81) + (2 * 9.81) * ACC_RANGE
// Initialize sensors library
MPU6050 imu;
int16_t ax, ay, az;
BMP085 barometer;

On the setup function, initiate the IMU and set the off-set values and range:

// Initialized sensors physically
Serial.println("Initializing I2C devices...");
Wire.begin();
imu.initialize();
barometer.initialize();
delay(10);
Serial.println("Testing device connections...");
Serial.println(barometer.testConnection() ? "BMP085 connection successful" : "BMP085 connection failed");
// Set IMU (gyrometer) offset for error evaluation
imu.setXAccelOffset(-4732);
imu.setYAccelOffset(4703);
imu.setZAccelOffset(8867);
imu.setXGyroOffset(61);
imu.setYGyroOffset(-73);
imu.setZGyroOffset(35);
imu.setFullScaleAccelRange(ACC_RANGE);

On the original code, you have this line:

IMU.readAcceleration(buffer[ix], buffer[ix + 1], buffer[ix + 2]);

Change it with the following lines of code:

imu.getAcceleration(&ax, &ay, &az);
    buffer[ix + 0] = ax;
    buffer[ix + 1] = ay;
    buffer[ix + 2] = az;

Then swap the order of these 2 blocks of code:

buffer[ix + 0] *= CONVERT_G_TO_MS2;
buffer[ix + 1] *= CONVERT_G_TO_MS2;
buffer[ix + 2] *= CONVERT_G_TO_MS2;
for (int i = 0; i < 3; i++)
{
    if (fabs(buffer[ix + i]) > MAX_ACCEPTED_RANGE)
    {
        buffer[ix + i] = ei_get_sign(buffer[ix + i]) * MAX_ACCEPTED_RANGE;
    }
}

Now the program is ready to be flashed to your device.

An error may occur while flashing, to fix this problem, you should switch off ESP NN acceleration.

To do that, locate ei_classifier_config.h in the library folder: /scr/edge-impulse-sdk/classifier/:

Find the line with #define EI_CLASSIFIER_TFLITE_ENABLE_ESP_NN 1, and change it from 1 to 0, you may want to open this file using Notepad:

#ifndef EI_CLASSIFIER_TFLITE_ENABLE_ESP_NN
    #if defined(ESP32)
        #include "sdkconfig.h"
        #define EI_CLASSIFIER_TFLITE_ENABLE_ESP_NN      0
        #define ESP_NN                                  1
    #endif // ESP32 check
    #if defined(CONFIG_IDF_TARGET_ESP32S3)
        #define EI_CLASSIFIER_TFLITE_ENABLE_ESP_NN_S3      1
    #endif // ESP32S3 check
#else
    #define ESP_NN                                  1
#endif

Save this file and try flashing your program again, this time, everything should work perfectly fine. You can open your serial monitor and see the result.

For further information on the configuration of the program file, you can refer to mjrobot post

Alert System

After data is processed and the result is recorded on the board, it will be transferred to an alert device on land via ESPNow wireless protocol.

Demo Picture

The device will be put on the back of the user when the user swims and the data will be collected and alerted accordingly.

The direction of future development

• Use lower frequency waves to process data as well as make the product smaller with better resistance to the environment.
• Add physical buttons to fix misidentification and make it able to connect to an app that can notify lifeguards instead of having to check frequently, and provide advice and instructions about artificial respiration when seeing a drowning person.

Conclusion

This prototype can be used by not only any disabled person but anyone learning to swim, children, elderly people, and even healthy experienced swimmers.

This is a critical life-saving device that could potentially save a person from drowning but still, there are a few advantages and disadvantages of the product that can be mentioned:

Pros:

• Successfully collected and accurately identified up to 85.7% of data samples about users' underwater actions.
• Data processing speed is fast enough for warnings.
• Helps rescue workers work more effectively with just a phone.
• Affordable price (about $5 to make)

Cons:

• There is still a small chance of misidentification
• Still relies on lifeguards with another device like a phone to receive notification that someone is drowning.

References

https://www.hackster.io/mjrobot/exploring-machine-learning-with-the-new-xiao-esp32s3-6463e5

https://randomnerdtutorials.com/esp-now-esp32-arduino-ide/

https://www.hackster.io/HackingSTEM/stream-data-from-arduino-into-excel-f1bede