4 wheels RC toy with object recognition model on board

Hey everyone and welcome to my page!

I want to share with you my project regaridng DIY remote control car with implementation of machine learning model on board, which may give to it autonomous features.The very next creature is right here a little bit bellow. I was inspired with newcoming technologies, robots, drones and desided to do, to study doing it, something related to this area.An idea has come to build something with remote control, which might either fly or drive with possibility to broadcast video and detect objects. I have used Raspberry board+webcam as image process unit, which detect objects. Arduino Mega 2560 with radiomodule as hardware core to stear the car. Video transmitiing is established via SSH streaming at this moment.So, let's go forward!

Structure of project:

• - openCV, ML model and image processing
• -nRF24L01+ radio module
• - Get on wheels
• - RPi4 Arduino communication

OpenCV, ML model and image processing

A great oportunity to process images, videos, streams and so on gives us OpenCV library. I gess everyone has noticed a green frame which looks for faces in our mobile's camera. It is a bright example of implementation of it into mobile software. It takes start from deep 90th and has been progressing all this years. OpenCV works together with neural webs, machine learning models depending in its aim. From faces recognition systems to medical cancer detection models. It sounds cool, is it?OpenCV lib is available for most of programming langages and has a great support community.How does it work under hood? In few words, each image can be represented as an array of pixels, which has values from 0 to 255, or multidementional array if it is colored with rgb scale.Video actually is a set of images, which are structured and being shown 24 per sec. OpenCV can treat it within iteration loops.Key moment is usage neural webs/ML models with OpenCV. For example, count people on photo of crowded place. It miight be easy if we have 20 people, but what if it is a photo from drone and there is countless amount of them? Here ML models can help you. Tensorflow, pythorch, utralistycs yolo models already have a trained models for object recognition and clasification. Take a look on this picture:

And here after yolov5 model usage:

It can detect and classify people even on the right side with 0.44 probability.

The main features of all those libraries are the opportunity to train your own model with your specific demands.

I have tried to use TensorFlow, Ultralytics Yolo, and Edge Impulse, and I cannot recommend the first one due to poorly organized documentation and package support. For me, a beginner in this area, it looks messy.

Ultralytics Yolo looks much better.

It is worth to highlight here the topic of ML for edge implementation. Here there is a possibility to make our model work on edge devices such as microcontrollers, mobiles, embedded systems, and so on, which possess extremely low RAM memory and CPU performance compared with our PC.

Firstly, aiming specifically for edge devices, I tried to train a custom model with TensorFlow, because it has a specially designed tool to withdraw a lite version of models for edge devices, and I encountered a lot of obstacles with incompatibilities with the system, dependencies, etc.

Ultralytics has the possibility to convert an ML model to any format you wish, let alone.tflite. But deploying it on Raspberry Pi 4, I could only get 1 frame per sec using a USB web cam, which looks really slow.

Edge Impulse has really helped me here. Their approach gives me a custom model with 15 times higher performance!

Well-organized documentation on their website and on YouTube.

Possibility to deploy your model as a zip library on Arduino or build it directly into your RPi board from the Edge Impulse web server! Amazing.

Once you have downloaded it, you will get a generated Python SDK package with Python scripts, which we can use in our projects.

The guide how to train ML model with Edge Impulse is well described in their documentation:

https://docs.edgeimpulse.com/docs/tutorials/end-to-end-tutorials/computer-vision/object-detection/detect-objects-using-fomo

How to deploy it on raspberry Pi board is described there as well:

https://docs.edgeimpulse.com/docs/edge-ai-hardware/cpu/raspberry-pi-4

There are a lot of sources how to train a custom model with other frameworks/libraries such as TensorFlow, Yolo and so on.

nRF24L01 radio module

It may bring you a little bit of pain in the back. Or maybe more than a little if you are not prepared for that.

It is the cheapest option on the market, but has a couple of moments which might drive you crazy.

To avoid all problems with establishing communication between modules, I highly advise you to study the nRF24 documentation here:

https://nrf24.github.io/

And if you are still struggling with troubles, you have to:

• solder a capacitor around 50 µF to the VCC and GND pins directly on module.
• check if the module works, because there are a lot of modules which are broken directly from production line.

Use this code to check if your module can scan the radio environment at all:

#include <SPI.h>
#include "nRF24L01.h"
#include "RF24.h"
#include "printf.h"
//
// Hardware configuration
//
// Set up nRF24L01 radio on SPI bus
RF24 radio(22, 23); //Arduino mega 2560, pins 9 & 10 for Arduino Nano
//
// Channel info
//
const uint8_t num_channels = 128;
uint8_t values[num_channels];
//
// Setup
//
void setup(void)
{
//
// Print preamble
//
Serial.begin(9600);
Serial.println("Scanner Air On");
printf_begin();
//
// Setup and configure rf radio
//
radio.begin();
radio.setAutoAck(false);
// Get into standby mode
radio.startListening();
radio.printDetails();
delay(5000);
// Print out header, high then low digit
int i = 0;
while ( i < num_channels )
{
printf("%x", i >> 4);
++i;
}
printf("\n\r");
i = 0;
while ( i < num_channels )
{
printf("%x", i & 0xf);
++i;
}
printf("\n\r");
}
//
// Loop
//
const int num_reps = 100;
void loop(void)
{
// Clear measurement values
memset(values, 0, sizeof(values));
// Scan all channels num_reps times
int rep_counter = num_reps;
while (rep_counter--)
{
int i = num_channels;
while (i--)
{
// Select this channel
radio.setChannel(i);
// Listen for a little
radio.startListening();
delayMicroseconds(512);
radio.stopListening();
// Did we get a carrier?
if ( radio.testCarrier() )
++values[i];
}
}
// Print out channel measurements, clamped to a single hex digit
int i = 0;
while ( i < num_channels )
{
printf("%x", min(0xf, values[i] & 0xf));
++i;
}
printf("\n\r");
}

and if your module in working condition it has to show something like that in serial monitor:

Each line represents a scan loop. Going from left to right, the digits correspond to each channel. Zeroes mean the channel is free from home radio emitters such as WiFi, mobile, and so on.

If every line is in a zero representation - these are bad news. Try to choose a free radio channel to reduce noise and interference, for example, WiFi.

On wheels

Before building my car, I had been developing my ideas on how to make it a reality.

I followed these thoughts:

Raspberry Pi 4 retrieves video from a USB webcam, processes it, and sends data to Arduino with information to move or not to move, and so on.

At the same time, Arduino has 2 modes: remote-controlled mode and autonomous mode.

Remote controlled mode implies waiting for data from the nRF24L01 module, keeping movement values x and y as 0 if data from the transmitter is in a neutral position (joysticks are untouched and output 512 out of 1023 in digital representation).

If the radio module is powered off, Arduino switches to autonomous mode and tries to turn around. There is a probability to catch a target in the camera view, and then if it happens, it will stop turning around and move toward the target (ball).

Prebuilt vehicle platforms are available on web market places. For example:

The thing is, I was not sure about how to embed raspberry Pi and arduino board with battery, how to make wheels to drive, consequently was decided to build it on my own from couple of detals from aliexpress and and woody boards

And wild horse is ready to go being totally equiped:

To control this toy is being used DIY remote controller:

Two Arduino joysticks, an Arduino Nano board, and an NRFL01 radio module have been used here. The wiring schematic is as follows:

Code for transmitter is here:

// Arduino pin numbers
#include "SPI.h"
#include "RF24.h"
#include "nRF24L01.h"
#define CE_PIN 9
#define CSN_PIN 10
#define INTERVAL_MS_TRANSMISSION 250
RF24 radio(CE_PIN, CSN_PIN);
const byte address[6] = "00001";
const int SW_pin = 2; // digital pin connected to switch output
const int X_pin = 3; // analog pin connected to X output
const int Y_pin = 4; // analog pin connected to Y output
int x;
int y;
struct payload {
int x;
int y;
};
payload payload;
void setup() {
pinMode(SW_pin, INPUT);
digitalWrite(SW_pin, HIGH);
Serial.begin(9600);
radio.begin();
//Append ACK packet from the receiving radio back to the transmitting radio
radio.setAutoAck(false); //(true|false)
radio.setChannel(0x6f);
radio.setCRCLength(RF24_CRC_8);
//Set the transmission datarate
radio.setDataRate(RF24_1MBPS); //(RF24_250KBPS|RF24_1MBPS|RF24_2MBPS)
//Greater level = more consumption = longer distance
radio.setPALevel(RF24_PA_MAX); //(RF24_PA_MIN|RF24_PA_LOW|RF24_PA_HIGH|RF24_PA_MAX)
//Default value is the maximum 32 bytes
radio.setPayloadSize(sizeof(payload));
//Act as transmitter
radio.openWritingPipe(address);
radio.stopListening();
}
void loop() {
payload.x = analogRead(X_pin);
payload.y = analogRead(Y_pin);
radio.write(&payload, sizeof(payload));

//to see what is going on with data just uncommend bellow
/*Serial.print("x:");
Serial.println(payload.x);
Serial.print("y:");
Serial.println(payload.y);
Serial.println("*---------------------*");
Serial.print("Switch:  ");
Serial.print(digitalRead(SW_pin));
Serial.print(" | ");
Serial.print("X-axis: ");
Serial.print(analogRead(X_pin));
Serial.print(" | ");
Serial.print("Y-axis: ");
Serial.print(analogRead(Y_pin));
Serial.println(" | ");*/
delay(50);
}

Code for car does not look so straidforward as previous ones.The thing is I should have connected and combined raspberry pi4 and arduino mega.The idea works as follows:RPI4 uses usb-webcam to process incoming video with opencv lib. Than, the detected object's coordinates has to be passed to arduino boards via uart port. In that same time, 2 servos are directly connected to rpi4 gpio pins with the purpose of aiming at the detected object.Wiring for car looks following:

As far as we know RPI4 board is very sensitive for power consumption, thus was decided to use separate power supply. It worth to be mentioned, the logic level shifter between rpi4 and arduino board using uart port, as logic voltage incompatibility between those boards can damage rpi4.

Arduino scetch for vehicle looks following:

#include "SPI.h"
#include "RF24.h"
#include "nRF24L01.h"
#include <Servo.h>
#define CE_PIN 22
#define CSN_PIN 23
t
const int X_pin = 0;
const int Y_pin = 1;
//--------ultrasonic sensor pins --------
int trigPin = 10;
int echoPin = 11;
const int engineA = 2;
const int engineB = 7;
const int right_engine_in1 = 3;
const int right_engine_in2 = 4;
const int right_engine_in3 = 5;
const int right_engine_in4 = 6;
const byte address[6] = "00001";
unsigned long previousMillis = 0;
const long time_interval = 3000;  // 2 second
long duration, inches;
int cm;
bool flag = false;
RF24 radio(CE_PIN, CSN_PIN);
Servo myservo;
int x;
int y;
int drive;
int pos;
unsigned long lastRadioTime = 0;
const unsigned long RADIO_TIMEOUT = 1000;
struct payload {
int x;
int y;
};
// does not work with automomous func implementation
// struct rpi_data {
//   int x_axes;
//   int y_axes;
// };
payload payload;
//rpi_data rpi_package;
void setup() {
myservo.attach(8);
//--------initialize a radio module -----------------
radio.begin();
//Append ACK packet from the receiving radio back to the transmitting radio
radio.setAutoAck(false); //(true|false)
radio.setChannel(0x6f);
radio.setCRCLength(RF24_CRC_8);
//Set the transmission datarate
radio.setDataRate(RF24_1MBPS); //(RF24_250KBPS|RF24_1MBPS|RF24_2MBPS)
//Greater level = more consumption = longer distance
radio.setPALevel(RF24_PA_MIN); //(RF24_PA_MIN|RF24_PA_LOW|RF24_PA_HIGH|RF24_PA_MAX)
//Default value is the maximum 32 bytes
radio.setPayloadSize(sizeof(payload));
//Act as receiver
radio.openReadingPipe(1, address);
radio.startListening();
//--------- engines pins initialisation ----------
pinMode(engineA, OUTPUT);
pinMode(right_engine_in1, OUTPUT);
pinMode(right_engine_in2, OUTPUT);
pinMode(engineB, OUTPUT);
pinMode(right_engine_in3, OUTPUT);
pinMode(right_engine_in4, OUTPUT);
//digitalWrite(SW_pin, HIGH);
digitalWrite(right_engine_in1, LOW);
digitalWrite(right_engine_in2, LOW);
digitalWrite(right_engine_in3, LOW);
digitalWrite(right_engine_in4, LOW);
pinMode(trigPin, OUTPUT);
pinMode(echoPin, INPUT);
Serial.begin(9600);
Serial1.begin(9600);
}
void loop() {
if (radio.available()) {
radio.read(&payload, sizeof(payload));
int *x_ptr = &x;
int *y_ptr = &y;
*x_ptr = payload.x;
*y_ptr = payload.y;
Serial.println("Received radio");
Serial.println(radio.available());
Serial.print("payload.x ");
Serial.println(payload.x);
Serial.print("payload.y ");
Serial.println(payload.y);
lastRadioTime = millis();
}
// --- Fallback to RPi if radio is lost ---
if (millis() - lastRadioTime > RADIO_TIMEOUT) {
delay(50);
data_from_rpi();
Serial.println("Switched to RPi autonomy");
}
//--------convert data into dergees of wheels -----------
float y_value = (float) y;
float angle_value = -1*(503-y_value)/1023*180;
int angle_pos = 90 - round(angle_value);
//Serial.print(angle_pos);
int distance = distance_sensor();
if (distance < 10){
x = 1023;
y = 0;
}
/*------------ moving strait -----------------*/
if (x > 522){
drive = map(x, 522, 1023, 0, 255);
analogWrite(engineA, drive);
digitalWrite(right_engine_in1, LOW);
digitalWrite(right_engine_in2, HIGH);
analogWrite(engineB, drive);
digitalWrite(right_engine_in3, HIGH);
digitalWrite(right_engine_in4, LOW);
}
/*---------------moving back-------------------*/
else if (x < 510){
drive = map(x, 500, 0, 0, 255);
analogWrite(engineA, drive);
digitalWrite(right_engine_in1, HIGH);
digitalWrite(right_engine_in2, LOW);
analogWrite(engineB, drive);
digitalWrite(right_engine_in3, LOW);
digitalWrite(right_engine_in4, HIGH);
}
/*----------------neutral----------------------*/
else{
digitalWrite(right_engine_in1, LOW);
digitalWrite(right_engine_in2, LOW);
digitalWrite(right_engine_in3, LOW);
digitalWrite(right_engine_in4, LOW);
}
// Serial.print("x:");
// Serial.print(payload.x);
// Serial.print("; ");
// Serial.print("y:");
// Serial.print(payload.y);
// Serial.print("; ");
// Serial.print("pos: ");
// Serial.println(angle_pos);
myservo.write(angle_pos);
radio.flush_rx();
Serial.print("distance from us sensor: ");
Serial.println(distance);
delay(50);
}
//-----------retrieve data from RPI ------------
// Python script on rpi4 process incoming webcam data with opencv
// and pretrained object detection model on Edge Impulse
// sends data into arduino via serial1
void data_from_rpi (){
// if frame data are not available, objects are not detected,
// car turns around
bool *flag_ptr = &flag;
unsigned long startTime = millis();
while (!Serial1.available()) {
if (millis() - startTime > 3000) {  // Timeout after 2 seconds
Serial.println("autonomous");
if (*flag_ptr==true){
x = 100;
y = 1024;
*flag_ptr = false;
} else {
x = 900;
y = 0;
*flag_ptr = true;
}
return;
}
// if serial data are available
if (Serial1.available()) {
//Serial1.readBytes((char*)&rpi_package, sizeof(rpi_package));
String data = Serial1.readStringUntil('\n');
float horizontal_axes = data.substring(0, data.indexOf(',')).toFloat();
float vertical_axes  = data.substring(data.indexOf(',') + 1).toFloat();
x = 100;//rpi_package.x_axes;
y = 1023.0 - horizontal_axes;
Serial.print("rpi_package.y_axes: ");
Serial.println(horizontal_axes);
Serial.print("Serial1 data: ");
Serial.print(data);
Serial.println(Serial1.available());
//Serial.print("Serial rpi is available cm: ");
//Serial.println(cm);
}
}
// -----------ultrasound distance sensor presettings ----------
int distance_sensor(){
digitalWrite(trigPin, LOW);
delayMicroseconds(5);
digitalWrite(trigPin, HIGH);
delayMicroseconds(10);
digitalWrite(trigPin, LOW);
duration = pulseIn(echoPin, HIGH);
// Convert the time into a distance
cm = (duration/2) / 29.1;     // Divide by 29.1 or multiply by 0.0343
inches = (duration/2) / 74;   // Divide by 74 or multiply by 0.0135
//Serial.print("cm: ");
//Serial.println(cm);
return cm;
}

On raspberry Pi side, the code is being executed in python environment with opencv library together with a pretrained machine learning model. As was mentioned above, Python code is being generated during machine learning model downloading from edge impulse server. One thing to note is that, we have to modify it according to our needs. In my situation, I had to extract x, y coordinates values of object to pass it via uart to arduino board.

Python script generated with Edge Impulse bellow. I just have added a function which passes values to servo :

#!/usr/bin/env python

#import device_patches       # Device specific patches for Jetson Nano (needs to be before importing cv2)

import cv2
import os
import sys, getopt
import signal
import time
from edge_impulse_linux.image import ImageImpulseRunner
import struct
import serial
from gpiozero import Servo
from time import sleep
import math
from gpiozero.pins.pigpio import PiGPIOFactory
factory = PiGPIOFactory()

#-----------servo initialisation----------------------- 
servo_x = Servo(18, 
                    min_pulse_width = 0.0005, 
                    max_pulse_width = 0.0025,
                    pin_factory = factory
                    )

servo_y = Servo(23, 
                    min_pulse_width = 0.0005, 
                    max_pulse_width = 0.0025,
                    pin_factory = factory
                    )

#------------ asign servo func --------------------
def servos(x,y):
    if x/2 and y/2 >= 48:
        servo_x.value = -0.25*x/48
        servo_y.value = -0.25*y/48
    else:
        servo_x.value = -1*(0.25*x/48 - 0.25)
        servo_y.value = -1*(0.25*y/48 - 0.25)
    


runner = None
# if you don't want to see a camera preview, set this to False
show_camera = True
if (sys.platform == 'linux' and not os.environ.get('DISPLAY')):
    show_camera = False

def now():
    return round(time.time() * 1000)

def get_webcams():
    port_ids = []
    for port in range(5):
        print("Looking for a camera in port %s:" %port)
        camera = cv2.VideoCapture(port)
        if camera.isOpened():
            ret = camera.read()[0]
            if ret:
                backendName =camera.getBackendName()
                w = camera.get(3)
                h = camera.get(4)
                print("Camera %s (%s x %s) found in port %s " %(backendName,h,w, port))
                port_ids.append(port)
            camera.release()
    return port_ids

def sigint_handler(sig, frame):
    print('Interrupted')
    if (runner):
        runner.stop()
    sys.exit(0)

signal.signal(signal.SIGINT, sigint_handler)

def help():
    print('python classify.py <path_to_model.eim> <Camera port ID, only required when more than 1 camera is present>')

def main(argv):
    try:
        opts, args = getopt.getopt(argv, "h", ["--help"])
    except getopt.GetoptError:
        help()
        sys.exit(2)

    for opt, arg in opts:
        if opt in ('-h', '--help'):
            help()
            sys.exit()

    if len(args) == 0:
        help()
        sys.exit(2)

    model = args[0]

    dir_path = os.path.dirname(os.path.realpath(__file__))
    modelfile = os.path.join(dir_path, model)

    print('MODEL: ' + modelfile)

    with ImageImpulseRunner(modelfile) as runner:
        try:
            model_info = runner.init()
            # model_info = runner.init(debug=True) # to get debug print out
            print('Loaded runner for "' + model_info['project']['owner'] + ' / ' + model_info['project']['name'] + '"')
            labels = model_info['model_parameters']['labels']
            if len(args)>= 2:
                videoCaptureDeviceId = int(args[1])
            else:
                port_ids = get_webcams()
                if len(port_ids) == 0:
                    raise Exception('Cannot find any webcams')
                if len(args)<= 1 and len(port_ids)> 1:
                    raise Exception("Multiple cameras found. Add the camera port ID as a second argument to use to this script")
                videoCaptureDeviceId = int(port_ids[0])
            serial_communication = serial.Serial('/dev/ttyAMA0', 9600) #/dev/ttyAMA0 /dev/ttyS0
            camera = cv2.VideoCapture(videoCaptureDeviceId)
            ret = camera.read()[0]
            if ret:
                backendName = camera.getBackendName()
                w = camera.get(3)
                h = camera.get(4)
                print("Camera %s (%s x %s) in port %s selected." %(backendName,h,w, videoCaptureDeviceId))
                camera.release()
            else:
                raise Exception("Couldn't initialize selected camera.")

            next_frame = 0 # limit to ~10 fps here

            for res, img in runner.classifier(videoCaptureDeviceId):
                if (next_frame > now()):
                    time.sleep((next_frame - now()) / 1000)

                #print('classification runner response', res)
                #print('classification runner response', img)
                if "classification" in res["result"].keys():
                    print('Result (%d ms.) ' % (res['timing']['dsp'] + res['timing']['classification']), end='')
                    for label in labels:
                        score = res['result']['classification'][label]
                        print('%s: %.2f\t' % (label, score), end='')
                    print('', flush=True)
                elif "bounding_boxes" in res["result"].keys():
                    print('Found %d bounding boxes (%d ms.)' % (len(res["result"]["bounding_boxes"]), res['timing']['dsp'] + res['timing']['classification']))
                    for bb in res["result"]["bounding_boxes"]:
                        print('\t%s (%.2f): x=%d y=%d w=%d h=%d' % (bb['label'], bb['value'], bb['x'], bb['y'], bb['width'], bb['height']))
                        img = cv2.rectangle(img, (bb['x'], bb['y']), (bb['x'] + bb['width'], bb['y'] + bb['height']), (255, 0, 0), 1)
     #----------------------------- coordinates yield-----------------------------
                
                data = res['result']
                bb = data['bounding_boxes']
                if len(bb) != 0 and bb[0]['value']>0.8:
                    
                    x = bb[0]['x']
                    y = bb[0]['y']
                    servos(x, y             
                    serial_communication.write(f"{x*1023/96},{y*1023/96}\n".encode())
                if (show_camera):
                    cv2.imshow('edgeimpulse', cv2.cvtColor(img, cv2.COLOR_RGB2BGR))
                    if cv2.waitKey(1) == ord('q'):
                        break
                        serial_communication.close()
                next_frame = now() + 100
        finally:
            if (runner):
                runner.stop()

if __name__ == "__main__":
   main(sys.argv[1:])

here is a video how it works in reality:

and here how it looks like inside on raspberry pi console:

Resume:

• It is possible to deploy an ML model for object recognition on a Raspberry Pi 4 device.
• Edge Impulse gives the best performance compared to others. The disadvantage is that it is a paid service for larger training tasks.
• Even though I have gotten relatively good scores with FPS on the RPi4, it remains slow with latency for larger tasks.
• To develop project into better perfomance way would be good point to change platform to one with gpu or npu.

link to my GitHub page is here:https://github.com/tech-science-club/RC_car_with_camera_and_ML_model

Update on 11.05.2025

Was decided to move forvard and add some features.
Idea to boost my RPi4 had not been leaving me, so I desided to explore ways how to boost its perfomance.
I chose a coral tpu among others alternatives.
It increases perfomance in above 5 - 10 times. So it is possible to use tensorflow and YOLO models directly on RPi4.
I have added fpv video transmitter and video reciever. I have gain possibility to process video stream from fpv cam online with Python in Pycharm IDE.

video is here: