Published September 1, 2022 © GPL3+

E-Nanny: ESP32 based smart baby sleep monitoring system

Fed-up of sleepless nights because of your cute little baby. E-nanny will put your babe back to sleep even before you wakeup.

IntermediateFull instructions provided6 hours3,734

E-Nanny: ESP32 based smart baby sleep monitoring system

Things used in this project

Hardware components

Espressif ESP32

INMP441 I2S Microphone

Software apps and online services

Audacity

Edge Impulse Studio

PlatformIO IDE

Hand tools and fabrication machines

Soldering iron (generic)

Solder Wire, Lead Free

Story

Introduction

If you’ve been a parent for longer than 24 hours, you know the expression “sleep like a baby” is just another lie. Babies don’t sleep soundly, like, at all. They grunt and groan, cough and sigh, scooch and squirm and wiggle. Everything is fun and games until your sweet little one wakes up crying in the middle of night on a daily basis. Babies have all the liberty to wake-up and sleep whenever they want, but often their parents don't have this. This is where E-nanny comes to rescue.

Working

E-nanny is based on a Keyword Spotting (KWS) algorithm. I know what you are thinking right now. Isn't an audio classification problem rather than KWS. I also thought the same first, but it seems KWS give more accuracy. Perhaps babies cry have similar rhythm of our natural languages. Once the "event" is detected the device will initiate a series of "counter measures" to put the baby back to sleep before the parent wake up. Beware! If the baby wakes up due to hunger or say dry napkin there is nothing my device can do and you'll have to put him back to sleep yourself. But babies more often due to far silly reasons, say a nightmare, so the device will be working in most cases.

Step 1: Collecting Data

For the Audio corpus I have completely relied on opensource sound databases like FreeSound. All audio I have used is copyright free and will be available in my GitHub page. I have collected 3 types of audio. They are:

1. Cry

2. Cough

3. Noise (which I recorded my self)

All in all, I am having around 15 mins of data. (10 mins cry, 5 mins noise, and a miniscule amount of cough)

Step 2: Cleaning Data

I am using Audacity for cleaning. Since I have mainly used free audio that was recorded by someone else I made sure that no unwanted sound is included and cropped it whenever I found such cases. Also leaving long pauses between crying (babies cry like that) will create confusion for the NN. So I have also removed long pauses if anywhere present. Amplify the audio if the amplitude is weak. Then I have exported all the audio into wav files at 16kHz sampling rate (16kHz is required by Edgeimpulse for KWS). Finally I renamed the audios in the format "label.number.wav" where label can be Cry, Cough or Noise. This is done to make labelling easy in the future inside edge impulse.

Before:

Before cleaning

After:

After cleaning

Step 3: Edge Impulse

Now it's time for the interesting part, training your model. Since there is good amount of material on this part in Edgeimpulse site itself, I won't dive deep into explanation. If you are a beginner seeing this video will be helpful.

After creating a new project upload your audio corpus from last step.

Start by create a new impulse as following

create impulse

Now got MFE option and generate feature

Generate Feature

Now it's time to train our model

Training

Ok, training completed and our model looks promising as data is neatly clustered. The accuracy seems not great. The main issue is cough that have miniscule amount of data. I'll try to improve the accuracy on the next part of the project.

Finally, let's deploy our model. For that download it is an Arduino library.

Deploy

Step 4: Preparing Hardware and Firmware

I am using an Esp32 and I2S microphone. But any 32 bit microcontroller and microphone will work. For interfacing inmp441 with esp32 I have taken reference of atomic14's code.

In platformIO create a new project based on any ESP32 boards. After project is created you have to unpack the zip file you downloaded from Edgeimpulse to the lib folder. (My model is attached below). The final the code is also attached below.

E-nanny in Action

I am playing a random YouTube video of baby crying on my laptop when inference was running on ESP32. As ESP32 detects baby crying it turns on the on board LED (blue LED)

E-nanny in Action

#include <Arduino.h>
#include <driver/i2s.h>
#include <E-nanny_inferencing.h>

/*************** I2C Macros ***************/
// you shouldn't need to change these settings
#define SAMPLE_BUFFER_SIZE 512
#define SAMPLE_RATE 16000
// most microphones will probably default to left channel but you may need to tie the L/R pin low
#define I2S_MIC_CHANNEL I2S_CHANNEL_FMT_ONLY_LEFT
// either wire your microphone to the same pins or change these to match your wiring
#define I2S_MIC_SERIAL_CLOCK GPIO_NUM_32
#define I2S_MIC_LEFT_RIGHT_CLOCK GPIO_NUM_25
#define I2S_MIC_SERIAL_DATA GPIO_NUM_33

/*************** Other Macros ***************/
#define LED_BUILTIN 2

/*************** Global variables ***************/
uint8_t ret = -1;
int16_t raw_samples[SAMPLE_BUFFER_SIZE];
float features[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE];
int indx = 0;

// don't mess around with this
i2s_config_t i2s_config = {
    .mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX),
    .sample_rate = SAMPLE_RATE,
    .bits_per_sample = I2S_BITS_PER_SAMPLE_16BIT,
    .channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,
    .communication_format = I2S_COMM_FORMAT_I2S,
    .intr_alloc_flags = ESP_INTR_FLAG_LEVEL1,
    .dma_buf_count = 2,
    .dma_buf_len = 1024,
    .use_apll = false,
    .tx_desc_auto_clear = false,
    .fixed_mclk = 0};

// and don't mess around with this
i2s_pin_config_t i2s_mic_pins = {
    .bck_io_num = I2S_MIC_SERIAL_CLOCK,
    .ws_io_num = I2S_MIC_LEFT_RIGHT_CLOCK,
    .data_out_num = I2S_PIN_NO_CHANGE,
    .data_in_num = I2S_MIC_SERIAL_DATA};


void setup()
{
  // we need serial output for the plotter
  Serial.begin(115200);
  // start up the I2S peripheral
  ret = i2s_driver_install(I2S_NUM_0, &i2s_config, 0, NULL);
  if(ret) Serial.printf("[Error] %d",ret);
  ret = i2s_set_pin(I2S_NUM_0, &i2s_mic_pins);
  if(ret) Serial.printf("[Error] %d",ret);

  pinMode(LED_BUILTIN,OUTPUT);
  
}

void loop()
{
  // read from the I2S device
  size_t bytes_read = 0;
  i2s_read(I2S_NUM_0, raw_samples, sizeof(int16_t) * SAMPLE_BUFFER_SIZE, &bytes_read, portMAX_DELAY);
  int samples_read = bytes_read / sizeof(int16_t);
  
  for(int i=0; i<samples_read; i++)
  {
    features[i+indx] = float(raw_samples[i]);
  }
  indx += samples_read;
  if(indx >= EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE)
  {
    // start inferencing
    ei_impulse_result_t result;

    // create signal from features frame
    signal_t signal;
    numpy::signal_from_buffer(features, EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE, &signal);
    // run classifier
    EI_IMPULSE_ERROR res = run_classifier(&signal, &result, false);
    //ei_printf("run_classifier returned: %d\n", res);
    if (res != 0)
      return;

    //****************
    // print predictions
    ei_printf("Predictions (DSP: %d ms., Classification: %d ms., Anomaly: %d ms.): \n",
              result.timing.dsp, result.timing.classification, result.timing.anomaly);

    // print the predictions
    for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++)
    {
      ei_printf("%s:\t%.5f\n", result.classification[ix].label, result.classification[ix].value);
    }
    //*****************/

   if(result.classification[0].value>0.75) //Label Cry
    digitalWrite(LED_BUILTIN,HIGH);

   else if(result.classification[2].value>0.75)//Label Noise
    digitalWrite(LED_BUILTIN,LOW);
#if EI_CLASSIFIER_HAS_ANOMALY == 1
    ei_printf("anomaly:\t%.3f\n", result.anomaly);
#endif

    indx = 0;
  }
}

E-Nanny: ESP32 based smart baby sleep monitoring system

Things used in this project

Hardware components

Software apps and online services

Hand tools and fabrication machines

Story

Introduction

Working

Step 1: Collecting Data

Step 2: Cleaning Data

Step 3: Edge Impulse

Step 4: Preparing Hardware and Firmware

E-nanny in Action

Code

Esp32 platformio code

E-nanny trained model

Credits

nafihahmd

Comments

Embed the widget on your own site

E-Nanny: ESP32 based smart baby sleep monitoring system

E-Nanny: ESP32 based smart baby sleep monitoring system

Things used in this project

Hardware components

Software apps and online services

Hand tools and fabrication machines

Story

Introduction

Working

Step 1: Collecting Data

Step 2: Cleaning Data

Step 3: Edge Impulse

Step 4: Preparing Hardware and Firmware

E-nanny in Action

Code

Esp32 platformio code

E-nanny trained model

Credits

nafihahmd

Comments

Related channels and tags