Multi-Client MJPEG Streaming From ESP32

Background

I use Blynk IoT platform a lot. When one of my projects required a video stream I was very pleased to discover there is a video widget included with the Blynk App builder.

I was planning to use ESP32-CAM or ESP32-EYE module to stream video, only to discover that Blynk widget does not display a video stream from any of the Espressif's examples, due to, as I discovered, "chunked" HTTP response.

Neither Blynk widget, not VLC (one of my favorite players) like chunked streams.

Challenge

I knew for a fact that the GStreamer app can stream to VLC and Blynk widget, so I set off exploring how to get rid of chunks.

The result is an MJPEG Streaming Server sketch, closely mirroring what GStreamer is doing, which makes video stream work in VLC:

https://github.com/arkhipenko/esp32-cam-mjpeg

Streaming to more than one client

While looking at various video solutions, I noticed this comment line in the code:

// TODO: make this work for multiple clients

Hmm. A quick test revealed that indeed only one client can stream at a time, which was not what I had in mind for my IoT project.

Challenge #2 - multiple clients

I decided to employ FreeRTOS facilities to make a multi-client streaming server, namely a Queue to hold currently streaming clients in, Tasks to be able to service new connection requests while streaming, and semaphores to coordinate grabbing new frames and streaming existing ones.

The result is MJPEG Multi-client Streaming Server sketch, which can handle up to 10 connected clients (the default limit of WiFi connections on ESP32 I found out), while still being compatible with VLC and Blynk widgets (and browsers of course).

https://github.com/arkhipenko/esp32-cam-mjpeg-multiclient

This has been tested with ESP32-CAM modules and ESP-EYE module.

ESP-EYE has particularly impressive performance streaming to browsers with almost no noticeable delay.

The sketch takes advantage of PSRAM (if present, which on these systems it almost always is), and makes a quick copy of the camera buffer, thus allowing the camera to grab new images while previous ones are being served to connected clients.

Disconnected clients are dropped out of the processing queue, and so goes the circle of life.

Cherry on top

As I was working on the streaming server, I was using what seemed to be an old driver file for theOV2640 camera.

There is a new driver available here: https://github.com/espressif/esp32-camera

with methods containing words like set_pixformat,set_contrast, and so on, promising further control over the camera.

Hmm... can I use the new driver with my sketch?

Challenge #3 - using new drivers

Update as of 9/30/2024 - the PlatformIO version of the multi-client code has the most recent updates and performance improvements. It is also linked directly to the latest esp-camera repository.

The below process is for Arduino IDE only - and is prone to breaking. Please use the PlatformIO option. I was able to achieve 29 FPS with AI Thinker ESP32-CAM board with VGA resolution and 70% JPEG quality.

The ESP32-CAMERA solution has an elaborate subfolder structure. On top of things, the standard ESP32 Arduino core has an older version of ESP32-CAMERA included, which immediately led to multiple conflicts.

"What if I stop fighting with subfolders?" I thought.

Thankfully, all files in the ESP32-CAMERA solution have unique names, so dumping them all in the same folder was not a problem, and with very few modifications the MJPEG Multi-client Streaming Server was running on the latest Espressif drivers:

https://github.com/arkhipenko/esp32-mjpeg-multiclient-espcam-drivers

The README file explains how to update the sources and compile.

I am very impressed with ESP-EYE - the speed and quality considering the size of this thing are quite impressive!

Practical Application

For instance this project.

Enjoy!