Introduction
Solution
Connection Diagram
Edge Impulse
Arduino Setup and Sketch Compilation
Energy Consumption
M5 Display
Prototype Images
EPIC DEMO
Closing

EdOliver, Victor Altamirano, Alejandro Sanchez

Published April 15, 2022 © MIT

Edge Driving Monitor

Driving Monitor that tracks the driver's blind spots with a warning system created with an ESP32 CAM and powered by Edge Impulse.

AdvancedFull instructions provided14 hours1,745

Things used in this project

Hardware components

Adafruit ESP32 CAM

M5Stack M5StickC ESP32-PICO Mini IoT Development Board

Software apps and online services

Edge Impulse Studio

Arduino IDE

Hand tools and fabrication machines

3D Printer (generic)

Story

Proyect link: https://studio.edgeimpulse.com/public/46428/latest

DISCLAIMER: This application is used for demonstrative and illustrative purposes only and does not constitute an offering that has gone through regulatory review. There is no representation as to the accuracy of the output of this application and it is presented without warranty.

Our Trailer:

Full project, build instructions and un-edited videos below!

Introduction:

As a driver, it’s quite important for you to know where the blind spots are on your own vehicle as well as other drivers' vehicles. Knowing this will help protect you and those around you from an easily-avoidable accident (no one wants to get sideswiped, really).

Blind spots are the areas to the sides of your car that can’t be seen in your rear mirror or side mirrors.

Despite being very noticeable in trucks (1), cars also have this type of blind spots, which can generate up to 800, 000 accidents per year (2).

Solution:

We will create a system that will be able to detect small vehicles, pedestrians and motorcycles in the automobile’s blind spots.

Current Solutions:

Mercedes-Benz Attention Assist uses the car's engine control unit to monitor changes in steering and other driving habits and alerts the driver accordingly.
Lexus placed a camera in the dashboard that tracks the driver's face, rather than the vehicle's behavior, and alerts the driver if his or her movements seem to indicate sleep.
Volvo's Driver Alert Control is a lane-departure system that monitors and corrects the vehicle's position on the road, then alerts the driver if it detects any drifting between lanes.
Saab uses two cameras in the cockpit to monitor the driver's eye movement and alerts the driver with a text message in the dash, followed by a stern audio message if he or she still seems sleepy.

As you can see these are all premium brands and there is not a single plug and play system that can work for every car. This, is our opportunity as most cars in the road are not on that price range and do not have these systems.

Connection Diagram:

This is the connection diagram of the system:

Edge Impulse:

To be able to use Machine Learning in a microcontroller, you have to decide the framework to use, in this case the service offered by Edge Impulse (https://www.edgeimpulse.com/) was used to accelerate the development of the solution.

The first thing we must do is put forward what we want to detect with our model, in this case it would be cars, people and motorcycles. Therefore the first stage is to obtain the dataset to feed the model.

The project is public, so you can analyze the dataset and use it as you like:

https://studio.edgeimpulse.com/public/46428/latest

For my Impulse, I uploaded 1224 images in total, being 958 for train and 266 for test, with the labels Cars, Motorcycle and Pedestrians.

The next thing will be to configure the Impuse that is going to be created, this in order to be able to read it from the ESP32 camera and be able to send it to the neural network.

NOTE: DO NOT CHANGE THESE SETTINGS IF YOU ARE GOING TO REPLICATE THIS PROJECT ON AN ESP32 CAM.

Now we must go to the Image tab and generate the features of the model.

Parameters:

Features:

Now we must configure the neural network that is going to be trained with our data, in this case Edge Impulse uses the transfer learning technique to facilitate the training of the networks, so for this example it is convenient to use a MobileNetV2 0.05

Don't forget to press the Start Training button.

Remember that having a network that obtains values very close to 100% in training can be a sign of data overfitting, so always avoid this from happening, however for that we will have to test the model in the Model testing tab.

Finally, since we saw that the model works, it does not fall into overfitting or underfitting, we must choose the device where we are going to display the model for its use, since I use the model for ESP32 and I will use the Arduino IDE, I will select from the list that library.

In the section below we will be able to build and download the model, Edge Impuse recommends leaving the EON Compiler enabled, so we will just leave it like that, especially do not forget to use the Quantized Model option for better performance.

Now we will follow the instructions to add the project to Arduino IDE, you can use the version directly from Edge Impuse or use the version: https://github.com/altaga/Edge-Driving-Monitor/blob/main/ei-edgedrivermonitor-arduino-1.0.3.zip

Arduino Setup and Sketch Compilation:

Once we have the library downloaded, we will open the example in the Edge_Impulse_ESP32_Cam folder (https://github.com/altaga/Edge-Driving-Monitor/tree/main/Edge_Impulse_ESP32_Cam), once open we will add the.zip library as shown shown in the picture.

Once the library is added, we will select the correct board for which we are going to compile, in this almost the ESP32 CAM.

Already selected the board now if you select the serial port of the board and flash the code.

If the compilation succeeds we should see the following result in the IDE.

Energy Consumption:

The code tried to take care of the ESP32's power consumption as much as possible, so the considerations to keep the consumption to a minimum were:

Use an inference network like the one Edge Impulse displays.
Send the microcontroller to Deep sleep while it is not making the inference in the neural network, the deep sleep is generated by a timer and you can lower or raise it according to your needs, in my case the device falls asleep for 2 seconds between readings.

Here is an analysis of the energy consumed by the device in 10 min, I will speed up the video 10 times so that you can see the entire analysis in 1 min, this analysis was performed with the Nordic Power Profiler Kit II.

With this analysis we can extrapolate the energy consumption that would be 50mAh, the analysis data will be in the file: ppk-20210904T064657.csv for you to be able to analyze the power consumption with more detail.

M5 Display.

As part of the solution it was decided that to show the results of the device a screen or some device with a display was necessary, it is not necessary to use a screen if you do not require it, it could be only an LED and thus keep energy consumption even lower, but for demonstration purposes the M5core2 was used and the serial communication was used to send the results from the ESP32 to the M5core2.

ESP32 Cam: Get Image, Analyze and Sleep.

fb = esp_camera_fb_get();
if (!fb) {
  esp_deep_sleep_start();
}
else {
  Serial.println(classify());
  Serial.flush();
}
esp_sleep_enable_timer_wakeup(TIME_TO_SLEEP * uS_TO_S_FACTOR); // 2 seconds
esp_deep_sleep_start();

M5Core2: Display result.

String serial = Serial2.readString();
if (serial.indexOf("Start") != -1) {  // Filter Noise from ESP32 Cam Module Sleep
  flag = true;
} 
else if (flag) { // Convert Serial to Float Array
  serial = serial.substring(serial.indexOf("Star") + 4, serial.indexOf("Stop"));
  float array[3] = { serial.substring(0, serial.indexOf(",")).toFloat(), serial.substring(serial.indexOf(",") + 1, serial.indexOf(",", serial.indexOf(",") + 1)).toFloat(), serial.substring(serial.indexOf(",", serial.indexOf(",") + 1) + 1).toFloat() 
  };

Communication as indicated was done by serial from ESP32 (GPIO3 RX and GPIO1 TX) to PORTC (Serial2) of M5core2

I did some static tests on the device to make sure the network was working properly.

Car:

Pedestrian:

Motorcycle:

Prototype Images:

EPIC DEMO:

Closing

Development with the Edge Impulse platform was a lot easier that developing the same application with Tensorflow or any other framework. It represents the way forward with its no-code principles and makes Machine Learning a lot fater to develop in comparison. The only thing that we found was that certain microcontroller-camera combos record a very low quality image that even these frameworks are not able to use for ML. Apart from that we are hopeful for the future of these kind of applications as ML and Computer Vision become much more ubiquitous.

Schematics

Code

#include <Arduino.h>
#include <WiFi.h>
#include <EdgeDriverMonitor_inferencing.h>
#include "esp_timer.h"
#include "img_converters.h"
#define CAMERA_MODEL_AI_THINKER // Has PSRAM
#include "esp_camera.h"
#include "camera_pins.h"

// raw frame buffer from the camera
#define FRAME_BUFFER_COLS           240
#define FRAME_BUFFER_ROWS           240
#define CUTOUT_COLS                 EI_CLASSIFIER_INPUT_WIDTH
#define CUTOUT_ROWS                 EI_CLASSIFIER_INPUT_HEIGHT
#define PART_BOUNDARY "123456789000000000000987654321"
#define uS_TO_S_FACTOR 1000000 /* Conversion factor for micro seconds to seconds */
#define TIME_TO_SLEEP 2 /* Time ESP32 will go to sleep (in seconds) */

camera_fb_t * fb = NULL;
uint8_t * _jpg_buf = NULL;
camera_config_t config;

const int cutout_row_start = (FRAME_BUFFER_ROWS - CUTOUT_ROWS) / 2;
const int cutout_col_start = (FRAME_BUFFER_COLS - CUTOUT_COLS) / 2;

String classify();
void cameraSetup();
void r565_to_rgb(uint16_t color, uint8_t *r, uint8_t *g, uint8_t *b);
int cutout_get_data(size_t offset, size_t length, float *out_ptr);

void setup() {
  Serial.begin(115200, SERIAL_8E2);
  Serial.println("Start");
  cameraSetup();
  fb = esp_camera_fb_get();
  if (fb) {
    classify();
  }
  // Drop First Classification
  fb = esp_camera_fb_get();
  if (!fb) {
    esp_deep_sleep_start();
  }
  else {
    Serial.println(classify());
    Serial.flush();
  }
  delay(1);
  esp_sleep_enable_timer_wakeup(TIME_TO_SLEEP * uS_TO_S_FACTOR);
  esp_deep_sleep_start();
}

void loop() {
  // This loop never start
}

void cameraSetup() {
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = Y2_GPIO_NUM;
  config.pin_d1 = Y3_GPIO_NUM;
  config.pin_d2 = Y4_GPIO_NUM;
  config.pin_d3 = Y5_GPIO_NUM;
  config.pin_d4 = Y6_GPIO_NUM;
  config.pin_d5 = Y7_GPIO_NUM;
  config.pin_d6 = Y8_GPIO_NUM;
  config.pin_d7 = Y9_GPIO_NUM;
  config.pin_xclk = XCLK_GPIO_NUM;
  config.pin_pclk = PCLK_GPIO_NUM;
  config.pin_vsync = VSYNC_GPIO_NUM;
  config.pin_href = HREF_GPIO_NUM;
  config.pin_sscb_sda = SIOD_GPIO_NUM;
  config.pin_sscb_scl = SIOC_GPIO_NUM;
  config.pin_pwdn = PWDN_GPIO_NUM;
  config.pin_reset = RESET_GPIO_NUM;
  config.xclk_freq_hz = 20000000;
  config.pixel_format = PIXFORMAT_JPEG;
  if (psramFound()) {
    config.frame_size = FRAMESIZE_240X240;
    config.jpeg_quality = 10;
    config.fb_count = 2;
  } else {
    config.frame_size = FRAMESIZE_240X240;
    config.jpeg_quality = 12;
    config.fb_count = 1;
  }
  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK) {
    //Serial.printf("Camera init failed with error 0x%x", err);
    return;
  }
  sensor_t * s = esp_camera_sensor_get();
  if (s->id.PID == OV3660_PID) {
    s->set_vflip(s, 1); // flip it back
    s->set_brightness(s, 1); // up the brightness just a bit
    s->set_saturation(s, 0); // lower the saturation
  }
  s->set_framesize(s, FRAMESIZE_240X240);
}

void r565_to_rgb(uint16_t color, uint8_t *r, uint8_t *g, uint8_t *b) {
  *r = (color & 0xF800) >> 8;
  *g = (color & 0x07E0) >> 3;
  *b = (color & 0x1F) << 3;
}

int cutout_get_data(size_t offset, size_t length, float *out_ptr) {
  // so offset and length naturally operate on the *cutout*, so we need to cut it out from the real framebuffer
  size_t bytes_left = length;
  size_t out_ptr_ix = 0;

  // read byte for byte
  while (bytes_left != 0) {
    // find location of the byte in the cutout
    size_t cutout_row = floor(offset / CUTOUT_COLS);
    size_t cutout_col = offset - (cutout_row * CUTOUT_COLS);

    // then read the value from the real frame buffer
    size_t frame_buffer_row = cutout_row + cutout_row_start;
    size_t frame_buffer_col = cutout_col + cutout_col_start;

    uint16_t pixelTemp = fb->buf[(frame_buffer_row * FRAME_BUFFER_COLS) + frame_buffer_col];

    uint16_t pixel = (pixelTemp >> 8) | (pixelTemp << 8);

    uint8_t r, g, b;
    r565_to_rgb(pixel, &r, &g, &b);
    float pixel_f = (r << 16) + (g << 8) + b;
    out_ptr[out_ptr_ix] = pixel_f;

    out_ptr_ix++;
    offset++;
    bytes_left--;
  }

  // and done!
  return 0;
}

String classify() {
  ei_impulse_result_t result = { 0 };

  // Convert to RGB888
  //fmt2rgb888(fb->buf, fb->len, PIXFORMAT_RGB888, _jpg_buf);

  // Set up pointer to look after data, crop it and convert it to RGB888
  signal_t signal;
  signal.total_length = CUTOUT_COLS * CUTOUT_ROWS;
  signal.get_data = &cutout_get_data;

  // Feed signal to the classifier
  EI_IMPULSE_ERROR res = run_classifier(&signal, &result, false /* debug */);

  // Returned error variable "res" while data object.array in "result"
  if (res != 0) return "Error";

  // Print short form result data
  String returnResult = "";
  for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
    returnResult+=String(result.classification[ix].value);
    if(ix!= EI_CLASSIFIER_LABEL_COUNT - 1){
      returnResult+=String(",");
    }
  }
  return String("Star"+returnResult+"Stop");
}

#include <M5Core2.h>

#define RXD2 13
#define TXD2 14

bool flag = false;
bool batFlag = true;
String label[3] = { "Car", "Motorcycle", "Pedestrian" };

void disp(int battery);

void setup() {
  M5.begin();
  SD.begin();
  M5.Lcd.setTextSize(4);
  M5.Lcd.fillScreen(WHITE);
  Serial.begin(115200);
  Serial2.begin(115200, SERIAL_8E2, RXD2, TXD2);
}

// the loop routine runs over and over again forever:
void loop() {
  float batVoltage = M5.Axp.GetBatVoltage();
  int batPercentage = int(( batVoltage < 3.2 ) ? 0 : ( batVoltage - 3.2 ) * 100);
  disp(batPercentage);
  if (Serial2.available()) {
    String serial = Serial2.readString();
    if (serial.indexOf("Start") != -1) {  // Filter Noise from ESP32 Cam Module Sleep
      flag = true;
    } else if (flag) {
      serial = serial.substring(serial.indexOf("Star") + 4, serial.indexOf("Stop"));
      float array[3] = { serial.substring(0, serial.indexOf(",")).toFloat(), serial.substring(serial.indexOf(",") + 1, serial.indexOf(",", serial.indexOf(",") + 1)).toFloat(), serial.substring(serial.indexOf(",", serial.indexOf(",") + 1) + 1).toFloat() };
      int place = 0;
      Serial.println(array[0]);
      Serial.println(array[1]);
      Serial.println(array[2]);
      if (array[1] > array[0]) {
        if (array[2] > array[1]) {
          place = 2;
        } else {
          place = 1;
        }
      } else {
        if (array[2] > array[0]) {
          place = 2;
          
        }
      }
      if(place ==0){
M5.Lcd.drawJpgFile(SD, "/car.jpg", 0, 0);
      }
            else if(place ==1){
        M5.Lcd.drawJpgFile(SD, "/mot.jpg", 0, 0);
      }
else{
      M5.Lcd.drawJpgFile(SD, "/ped.jpg", 0, 0);  
      }
      Serial.println(label[place]);
      batFlag = true;
      flag = false;
    } else {
      flag = false;
    }
  }
}

void disp(int battery) {
static int batMem;
if (batMem != battery || batFlag) {
    M5.Lcd.setTextColor(WHITE);
    M5.Lcd.setCursor(20, 5);
    M5.Lcd.println("Battery:"+String(batMem) + "%");
    M5.Lcd.setTextColor(BLACK);
    batMem = battery;
    M5.Lcd.setCursor(20, 5);
    M5.Lcd.println("Battery:"+String(battery) + "%");
    batFlag = false;
  }
}