Remote control and Video Monitoring with ESP32 for Robot

ABOUT THIS PROJECT

The basic idea was to build a control module that can also enable image streaming. This should be a universal module that can easily replace already used modules. My current projects are building a remote control for car robots and RobotArm. The solution can allow swapping a Bluetooth model without having to change anything in the current software. The module was originally developed to control robots, but without any restrictions it can be used to control any other module. You’ll be able to control the robot using a web server that displays a video streaming of what the robot “sees”. You can control your robot remotely even if it’s out of your sight or even if the web server can only be reached via the internet. That doesn't bother function robot at all.

The principle is shown in the picture below, , you can find more here.. A mobile device communicates with the Arduino either via Bluetooth or via Wi-Fi. However, communication via Wi-Fi has the advantage that, in addition to the command, we also have a picture of the surroundings.

Wi-Fi is most common protocols that mobile devices have. Next we will build a project to make a mobile devices interact with ESP32 board over Wi-Fi. I will show how you can use different computers, which are responsible for controlling the wheels and for streaming and object or face detection. The simplest Arduino is responsible for motor functions. ESP32 regulates image streaming and image recognition. It can also generate controls for driving in "follow mode", which will be explained in another project.

Little "Big Brother watching you

About ESP32

The ESP32-CAM is a very small camera module that costs about 10 US dollars and is therefore comparable in price to the HC 05 Bluetooth module.

The ESP32-CAM is not a module but an independent ESP32 board with an integrated camera and SD card module. With the camera you can take pictures, record videos or even set up a live stream that can be accessed via the web browser. It is perfect for a low-budget surveillance camera. For good results with video streaming, I recommend using an ESP32-CAM with external antenna.

Besides the OV2640 camera, ESP has several GPIOs for connecting peripherals, which can be useful. In principle, you could build really good controlled robots with the module without using too many other components. Unfortunately, some pins can only be used to a limited extent, others should always be left unconnected. Such a construction without good "know how" would suffer from many problems. That's why I decided to use the ESP module only for communication and video streaming.

The Application can be extended to a versatile and general purpose Remote control.

The upload

The ESP32-CAM board is delivered without an ISP programmer (i.e. without a USB connection), so it is necessary to use a USB/UART converter to program the module. An FTDI converter with a USB Mini-B connector is the easiest way to program the ESP32-CAM board and used here. And best of all: You can easily program it with your Arduino IDE - just like your Arduino boards. However, you have to make a few preparations for this.

VCC and GND pin of ESP32 is connected with the VCC and GND pin of the FTDI board.

Tx of and Rx of ESP32 is connected with Rx and Tx of the FTDI board.

Note: For programming, GND must be connected to GPIO0 on the board (e.g. with a jumper). This bridging is then removed again after programming. Before each programming, the reset button (on the bottom of the board) of the board must also be pressed that is used to reset the program uploaded from the software.

In this example, we use Arduino IDE to program the ESP32-CAM board. So, you need to have Arduino IDE installed as well as the ESP32 add-on.

I am not going to explain the robot arm setup, Install Arduino IDE Libraries, Install Arduino IDE ESP32 board, the different boards and modules that I have used and the powering of the system, because I already did that in previous tutorials.

As you can see the ESP32 Cam is unable to connect with the Wi-Fi, this is just because the FTDI and Arduino are not able to provide enough current to the ESP32 Cam Module. Now to fix these types of errors we can use an external 5V regulated Power supply to power up the ESP32 Module. So I got myself a 5V regulated power supply board based on the LM7805 voltage regulator.

The live stream web server

Fortunately, the ESP32 also has many examples that you can use as a guide. We can also find our live stream web server in the examples. Navigate to the following point within the Arduino IDE:

"File → Examples → ESP32 → Camera → CameraWebServer" is now selected in the menu of the Arduino IDE, whereby a rather long sketch for creating a streaming server is loaded. In dies project with the ESP32, we connect the ESP32 to a wireless router. This way we can access the ESP32 through the local network. In this situation the router acts as an access point and the ESP32 is set as a station. In this scenario, you need to be connected to your router (local network) to control the ESP32. This has the advantage that robots can be reached via the Internet. Remote access to the home network via VPN and the appropriate setting in the router enable the robot to be controlled from anywhere. Then the robot is connected to a certain WLAN and for this you have to store a password for this WLAN in the program, which is not a particularly flexible solution.

If you want to operate the board independently of the router, you can also create an access point with the ESP32 and use it for video streaming. To do this, the following changes must be made to the sketch bellow:

There are still a few small adjustments to be made in this sketch:

There are only two program passages that we have to change. One is the camera model. We use the AI_THINKER camera model. Modify the comments so that #define CAMERA_MODEL_AI_THINKER takes effect.

Then, make sure you select the right camera module. In this case, we’re using the AI-THINKER Model.

#define CAMERA_MODEL_AI_THINKER

//#define CAMERA_MODEL_M5STACK_PSRAM

//#define CAMERA_MODEL_M5STACK_WITHOUT_PSRAM

Last but not least, the WLAN data must be entered. To do this, the following two lines must be adjusted:

HTML code and handle requests

The software flow of these steps:

• When web browser connects to web server, we send the index html.
• After loading the index html, web browser continue requesting video

In Arduino code in order to implementing video stream in a website esp32 cam we will use combined two codes C++ and HTMP. In order to do live stream using HTTP we will use the below HTTP format:

HTTP/1.1

Content-Type: multipart/x-mixed-replace; boundary=frame

--frame

Content-Type: image/jpeg

[image 1 encoded jpeg data]

--frame

Content-Type: image/jpeg

[image 1 encoded jpeg data]

We have to use the quotation marks to send this HTTP format.

In the index html just add the following line to the header area

<meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0;\" />\

This adapts the page to the display size.

I'm not giving step-by-step instructions here. In this case MJPEG stream boundary is fixed length, for split into a MJPEG stream of JPEG binary, each frame JPEG binary decode to RGB format. We have after to process HTTP header and response manually:

String index_html = " HTML code “

As the image was already in place, I used the following html code to create the HTML structure we would next style with CSS.

The required button for control is selected using the respective CSS selector. The background property can then be set to include a background image and change the image type as required. The border of the button can also be removed to show only the image itself.

Serial interface

All Arduino boards have at least one Serial (also known as UART or USART) interface. It communicates with the computer or Bluetooth module via digital pins 0 (RX) and 1 (TX) and via USB. Therefore, when using these functions, you cannot use pins 0 and 1 for digital input or output. Most novice Arduino coders use a specific baud rate value for all their programs and functions, most value of 9600 is used for communication with Bluetooth.

For ESP32 9600 baud is used here for communication with arduino. The real beauty of this solution is that almost any pin can be used as TX or RX pin for any serial port.

Which pins to use for the second serial port is not an easy decision. Some pins can only be used to a limited extent; others should always be left unconnected. Some pins send PWM signals when booting the ESP32 or briefly go HIGH. If you put something on there at boot time that pulls the pins HIGH or LOW, the unexpected can happen. Depending on what you attach to the pins, this can be annoying, but it doesn't have to be. I had problems with pins 12, 15, 16, 4. Finally I found pins 15 and 13 as the only solution that has no problems with booting.

The second serial port for communication with Arduino is defined as follows:

Serial2.begin(baud-rate, protocol, RX pin, TX pin);

#define RXD2 13 //Rx for serial2

#define TXD2 15 // Tx for serial2

Robot Controls

The website is presented in such a way that the image also has two by five panels that send characters, as shown in the image below.

In this case we used the right side to control the surveillance car and the left side to control the camera:

buttons to move the car in Left, Right, Forward and reverse directions as shown in the image above A- Forward, D -Backward, B-<Left, C->Right, and E-Stop

On other side buttons to move the camera: 1-Up, 4-Down, 2-buttons “O”, 3-“ S” and 5-“ L”.

For RobotArm in our example we introduced second mode as explained here. So it is possible to use the arm with 4 DOF ( "Degrees of Freedom"). Similar to app, there are two by five control buttons. Everyone sends own code.

Here HTTP communication protocol will be used to receive video streaming from the OV2640 camera in background over the web browser.

This web controlled surveillance car can be easily built using the WebServer.

Here you can adapt commands to your own module. As can be seen from the listing, command “strcmp” (function) starts comparing the first character of each string. If they are equal to each other, it continues with the following pairs until the characters differ or until a terminating null-character is reached.

As a result, the corresponding character is sent to the Serial output. The robot moves as long as you’re pressing the buttons. If you release any button, the robot will stop.

However, we’ve included the Stop button that can be useful in case the ESP32 doesn’t receive the stop command when you release a button.

Testing the code

After successfully uploading the sketch, the GND-GPIO0 connection is removed; the serial console is opened in the Arduino IDE and switched to 115200 baud. After pressing the reset button on the board, the IP for the streaming server should be displayed in the serial console after initialization.

Now any web browser can be opened with this IP as URL and the web interface of the ESP32-CAM streaming server should appear.

The raw video stream (a live image from the camera) can be reached at the URL 92.168.4.1:81/stream and should be able to be consumed by streaming programs.

On the picture below you can see how stream and command area are displayed on an Android. Slightly different display you see on PC device with Windows. The program is optimized for Android.

For the buttons, a color change is planned for only two buttons: Stop (red) and L (blue).

The program “startCameraserver” is placed in the setup because the program would be started several times in the loop. Web stream remains, but with an error message on the serial monitor: httpd: httpd_server_init: error in listen (112) ESP32

Only feedback from serial2 is placed in the loop. So we have in Serial Monitor von Arduino information, which commands receive controlled module.

Summary and Conclusion

In the recent century, Robotics is the most emerging field of research. Methods for remotely controlled car robots have already been described in various publications. There are some remote controlled Robots also that help us to carry out various operations. Efficient motion intent communication is necessary for safe and collaborative work environments with collocated humans and robots. Robots often have difficulty efficiently communicating their motion intent to humans. Therefore, a transmission image of the environment would be of great help to the one who controls the robot. New technology makes that possible without a large additional price. Controlling devices also becomes easier. With these changes, robots will be able to communicate wirelessly faster than ever. The relevant aspects for a solution based on ESP32 could be identified, compiled and formulated uniformly in this work.

The robotic arm control is more of a mechanical system control problem than a problem of controlling individual actuators of a robotic arm and a solution using video Stream would be appreciated, which is the focus of this article.

Previous tutorials you can find here:

Simple Follow-Me Function for 4WD Arduino car Kit:

https://create.arduino.cc/projecthub/paralija/simple-follow-me-function-for-4wd-arduino-car-kit-4215fd

use the GY-521 module (MPU-6050 Board) to make a 2WD robotcar drive in a straight

https://www.hackster.io/Kenan-Paralija/making-a-2wd-arduino-vehicle-drive-straight-ae40ee

RobDino -DIY Robot Arm

https://www.hackster.io/Kenan-Paralija/the-4-axis-movable-arm-with-remote-control-and-camera-c2610b