Smart Ethernet-enabled Person detector(RP2040, W5100S, TFLM)

1. Introduction

This project aims to demonstrate how to deploy an embedded person detection AI model to RP2040-based board and transmit the results via Ethernet connection.

The system consists of an Arducam 2MP SPI camera that is connected to the W5100S-EVB-Pico. The captured image will be serving an input to ML model that will detect a person in it. Then, the results will be transmitted to a local TCP server via Ethernet.

2. Hardware setup

As SPI0 is used for communication between RP2040 and W5100S, the Arducam 2MP SPI camera is connected to SPI1.

Optionally, USB-to-TTL can be connected to UART0 pins, if you wish to see camera output via PC.

3. Software setup

For this demo application I used person detection example code from Raspberry Pi pico-tflmicro repository.

Since I started this project before discovery of ArduCAM's fork of pico-tflmicro repository, I used some code from ArduCAM RPI-Pico-Cam repository.

Basically, the application is doing the same as original sample:

1. Capture image from the camera

2. Invoke detection on the image

3. Send the image via serial

4. Send the person detection score via serial

5. Send the person detection score via Ethernet to local TCP server

For Ethernet connectivity I ported WIZnet ioLibrary by referencing to RP2040-HAT-C projects.

Important note: as ArduCAM and WIZnet are using same variable for SPI pin definitions, it is required to change one of them.

As I am at beginning level, I did not proceed with TCP client implementation. Instead I used loopback client from WIZnet example and simply sent a customized message to the server.

How to build the application

As I didn't make any changes to the pre-trained model, I will not describe this part.

To build the source code, follow below steps:

1. Set up Pico SDK and required toolchain on your machine. Raspberry Pi has pretty cool guide here.

2. In my case I used static IP. Update according to your settings. Also update destination TCP server address and port.

3. Create a build directory and run cmake command

mkdir build
cd build
cmake ..

4. Press BOOTSEL button and connect W5100S-EVB-Pico to PC via USB

5. Copy the compiled firmware to the board

4. Testing

For testing I ran TCP server on WIZ510SR-RP module. The reason behind is simple, I wanted to send image from W5100S-EVB-Pico to the module, which then would convert it back to serial. However, I couldn't figure out how to transmit the image buffer.

In the below video, you can see person detection app being tested on Lego minifigure and Superman' printed image. The detected score is being transmitted to TCP server via Ethernet.

5. Possible Applications

This project serves as a practical demonstration, highlighting the seamless addition of network capabilities to existing detection system.

However, with added SSL it would be possible to create more advanced and complex real-time detection systems.

6. Conclusion

This project successfully demonstrated the integration of Ethernet connectivity with the Raspberry Pi Pico platform, utilizing TensorFlow Lite Micro for real-time person detection. By harnessing the computational power of the Pico's RP2040 microcontroller, the system efficiently processes person detection algorithms and transmits scores to a local TCP server, showcasing a practical application of IoT in monitoring and security. This proof of concept opens the door for further exploration and customization of the system for a variety of applications, from enhanced surveillance systems to smart building management.