Lesson 9: Make ESP32 React to Face Detection

In this tutorial we will build the most minimal example of face detection for ESP32 devices.

This will actually be easier than our last Lesson 8: Face Detection with your BallyBot where a web-server is streaming the facial detection footage. In creating this simplified version you will deepen your understanding on how the AI works. Minimally, we still need some indication the the face is detected so we are using the built in LED of the ESP32-CAM. In this case that is going to accessible in our code from io pin4.

Step 0: Adding Libraries

The First Step is to include all the libraries that will be used in this project. In this case we have the esp_camera.h for camera usage and 2 ml libraries for face detection:

#include "esp_camera.h"
#include "human_face_detect_mnp01.hpp"
#include "human_face_detect_msr01.hpp"

Step 1: Initialize LED

To initialize the LED we need to set the io4 pin that we are using to OUTPUT. If your ESP32-Cam is in a BallyBot you also should set the motor pin io14 as it can sometimes get read as HIGH by default.

void setup() {
  Serial.begin(115200);

  pinMode(4, OUTPUT);
  digitalWrite(4, HIGH); /*just to see it when uploaded*/
  pinMode(14, OUTPUT);
}

We also of course added Serial.begin(115200); so that we can read future Serial.print() commands.

Step 2: Initialize Camera

To initialize the Camera there are many settings that are not exactly relevant to most projects but still must be set. We add them in 2 steps:

• create defines
• add defines to config variable
• initialize camera with config

Here is the list of defines will create before our setup() function:

#define CAMERA_MODEL_AI_THINKER
#define PWDN_GPIO_NUM 32
#define RESET_GPIO_NUM -1
#define XCLK_GPIO_NUM 0
#define SIOD_GPIO_NUM 26
#define SIOC_GPIO_NUM 27

#define Y9_GPIO_NUM 35
#define Y8_GPIO_NUM 34
#define Y7_GPIO_NUM 39
#define Y6_GPIO_NUM 36
#define Y5_GPIO_NUM 21
#define Y4_GPIO_NUM 19
#define Y3_GPIO_NUM 18
#define Y2_GPIO_NUM 5
#define VSYNC_GPIO_NUM 25
#define HREF_GPIO_NUM 23
#define PCLK_GPIO_NUM 22

Next create a new function I will call CameraSetup() that will create a config variable and then fill it with our defines:

void CameraSetup() {
  camera_config_t config;
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = Y2_GPIO_NUM;
  config.pin_d1 = Y3_GPIO_NUM;
  config.pin_d2 = Y4_GPIO_NUM;
  config.pin_d3 = Y5_GPIO_NUM;
  config.pin_d4 = Y6_GPIO_NUM;
  config.pin_d5 = Y7_GPIO_NUM;
  config.pin_d6 = Y8_GPIO_NUM;
  config.pin_d7 = Y9_GPIO_NUM;
  config.pin_xclk = XCLK_GPIO_NUM;
  config.pin_pclk = PCLK_GPIO_NUM;
  config.pin_vsync = VSYNC_GPIO_NUM;
  config.pin_href = HREF_GPIO_NUM;
  config.pin_sscb_sda = SIOD_GPIO_NUM;
  config.pin_sscb_scl = SIOC_GPIO_NUM;
  config.pin_pwdn = PWDN_GPIO_NUM;
  config.pin_reset = RESET_GPIO_NUM;
  config.xclk_freq_hz = 10000000;
  config.pixel_format = PIXFORMAT_RGB565;
  /* init with high specs to pre-allocate larger buffers */
  config.frame_size = FRAMESIZE_QVGA;
  config.jpeg_quality = 40;
  config.fb_count = 2;
}

The last part of CameraSetup() is to call esp_camera_init(&config) and check for errors:

/* camera init, keep this in your CameraSetup */
  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK) {
    Serial.printf("Camera init failed with error 0x%x", err);
    return;
  }

One item you might wonder "why don't we just type the value of the defines into configs?". And the answer is: You can!

The reason why I do not in this tutorial, is because no other code online does this. Having your example code look similar enough to the other projects you encounter will make it much easier to handle them both.

Step 3: Capture Frames with Camera

Now that the initialization code is made we need to put it to use! That means calling the CameraSetup() function and running frame capture code in the loop.

To start lets modify Setup() to call CameraSetup() :

void setup() {
  Serial.begin(115200);

  pinMode(4, OUTPUT);
  digitalWrite(4, HIGH); /*just to see it when uploaded*/
  pinMode(14, OUTPUT);

  CameraSetup();
}

Then we need to add code in our Loop() to get and clear the new frames:

void loop() {
  camera_fb_t *fb = esp_camera_fb_get();
  if (!fb) {
    Serial.println("Camera capture failed");
    return;
  }

/* */

  esp_camera_fb_return(fb);
  delay(50);
}

Do note that this still does not give us any visible feedback if it is active yet. The way we can tell everything is working is lack of errors.

Step 4: Add Face Detection Code

This is the most important step. We need to make the face detection model infer if there is a face or not and then get it to print something if so. Fortunately it is not that hard after we have everything else working.To start we will look at the inference command and its inputs:

std::list<dl::detect::result_t> results = 
s1.infer((uint16_t *)fb->buf, {(int)fb->height, (int)fb->width, 3});

Look at is what is s1.infer returning std::list<dl::detect::result_t>, or in English a list of detection results. If you ctrl + click the result_t it will give you more info on what is really is storing.

Then the parameters for s1.infer are:

• fb->buf - the frame data
• {fb->height,fb->width, 3} - the frame dimensions

So it takes the info and returns a list of boxes for detected faces.Now with that understanding let's look at an implementation of it here:

std::list<dl::detect::result_t> results =
s1.infer((uint16_t *)fb->buf, {(int)fb->height, (int)fb->width, 3});
results = 
s2.infer((uint16_t *)fb->buf, {(int)fb->height, (int)fb->width, 3}, results);

if (!results.empty()) {
  Serial.println("Face detected!");
}

So we are taking results inferring with s1.infer, then running that through s2.infer() and then checking if those results are empty.To add this code into your loop place it in between getting the camera frame-buffer and returning it like this:

Upon running your code at this point, you should see your ESP32-Cam print "Face detected!" when it detects faces.

Step 5: Connect LED to Face Detection

Now that you have the face detection working all that is required here is to set up the if statements to toggle the light HIGH & OFF:

std::list<dl::detect::result_t> results =
s1.infer((uint16_t *)fb->buf, {(int)fb->height, (int)fb->width, 3});
results = 
s2.infer((uint16_t *)fb->buf, {(int)fb->height, (int)fb->width, 3}, results);

if (!results.empty()) {
  Serial.println("Face detected!");
  digitalWrite(4, HIGH);
} else {
  digitalWrite(4, LOW);
}

Step 6: Upload & Test

This section is assuming you are using the BallyBot's breakout Board to upload:

• Connect FTDI uploader
• Make sure Arduino IDE has AI Thinker ESP32-CAM board selected
• Select the correct port for the ESP32-CAM
• Click the Upload Button
• Turn the BallyBot upload switch up and then turn the power switch off -> on
• Once Arduino IDE finished upload turn BallyBot upload switch down and power off -> on

Here is the full code for reference:

#include "esp_camera.h"
#include "human_face_detect_mnp01.hpp"
#include "human_face_detect_msr01.hpp"

#define CAMERA_MODEL_AI_THINKER
#define PWDN_GPIO_NUM 32
#define RESET_GPIO_NUM -1
#define XCLK_GPIO_NUM 0
#define SIOD_GPIO_NUM 26
#define SIOC_GPIO_NUM 27

#define Y9_GPIO_NUM 35
#define Y8_GPIO_NUM 34
#define Y7_GPIO_NUM 39
#define Y6_GPIO_NUM 36
#define Y5_GPIO_NUM 21
#define Y4_GPIO_NUM 19
#define Y3_GPIO_NUM 18
#define Y2_GPIO_NUM 5
#define VSYNC_GPIO_NUM 25
#define HREF_GPIO_NUM 23
#define PCLK_GPIO_NUM 22

#define LED_GPIO_NUM 4  // Assuming LED is connected to GPIO 4

HumanFaceDetectMSR01 s1(0.1F, 0.5F, 10, 0.2F);
HumanFaceDetectMNP01 s2(0.5F, 0.3F, 5);

void setup() {
  Serial.begin(115200);
  Serial.setDebugOutput(true);

  pinMode(LED_GPIO_NUM, OUTPUT);
  digitalWrite(LED_GPIO_NUM, LOW);

  CameraSetup();
}

void loop() {
  camera_fb_t *fb = esp_camera_fb_get();
  if (!fb) {
    Serial.println("Camera capture failed");
    return;
  }

  std::list<dl::detect::result_t> results =
      s1.infer((uint16_t *)fb->buf, {(int)fb->height, (int)fb->width, 3});
  results = s2.infer((uint16_t *)fb->buf, {(int)fb->height, (int)fb->width, 3},
                     results);

  if (!results.empty()) {
    Serial.println("Face detected!");
    digitalWrite(LED_GPIO_NUM, HIGH);

  } else {
    digitalWrite(LED_GPIO_NUM, LOW);
  }

  esp_camera_fb_return(fb);
  delay(50);
}

void CameraSetup() {
  camera_config_t config;
  config.ledc_channel = LEDC_CHANNEL_0;
  config.ledc_timer = LEDC_TIMER_0;
  config.pin_d0 = Y2_GPIO_NUM;
  config.pin_d1 = Y3_GPIO_NUM;
  config.pin_d2 = Y4_GPIO_NUM;
  config.pin_d3 = Y5_GPIO_NUM;
  config.pin_d4 = Y6_GPIO_NUM;
  config.pin_d5 = Y7_GPIO_NUM;
  config.pin_d6 = Y8_GPIO_NUM;
  config.pin_d7 = Y9_GPIO_NUM;
  config.pin_xclk = XCLK_GPIO_NUM;
  config.pin_pclk = PCLK_GPIO_NUM;
  config.pin_vsync = VSYNC_GPIO_NUM;
  config.pin_href = HREF_GPIO_NUM;
  config.pin_sscb_sda = SIOD_GPIO_NUM;
  config.pin_sscb_scl = SIOC_GPIO_NUM;
  config.pin_pwdn = PWDN_GPIO_NUM;
  config.pin_reset = RESET_GPIO_NUM;
  config.xclk_freq_hz = 10000000;
  config.pixel_format = PIXFORMAT_RGB565;
  /* init with high specs to pre-allocate larger buffers */
  config.frame_size = FRAMESIZE_QVGA;
  config.jpeg_quality = 40;
  config.fb_count = 2;
  /* camera init */
  esp_err_t err = esp_camera_init(&config);
  if (err != ESP_OK) {
    Serial.printf("Camera init failed with error 0x%x", err);
    return;
  }
}

Troubleshooting:

• human_face_detect_mnp01.hpp not found:

This can happen if you downloaded a version of esp32 that is (>3.0.7 and <3.2.0). To fix this download the older version of the esp32 board manager 3.0.7

Conclusion:

Your BallyBot now interacts with its surroundings! This foundation enables:

• Security alerts: Log detections to SD card.
• Social behaviors: Add speech with MP3 modules.
• Navigation: Follow faces with motor control.

Next Up:Lesson 10: Face Tracking & Autonomous FollowingProgram BallyBot to pivot its camera or chassis to keep faces centered in view!

Resources

• Full Code for Lesson 9

Previous Lesson: Face Detection

• Previous Lesson: Face Detection

Let me know if you’d like help expanding reactions (e.g., adding voice, motors)! 🤖✨