360 Live Streaming + Object Detect(DPU) with KR260

We conducted real-time object detection on 360 live streaming image data.

For this, We used the RICOH THETA 360° camera.

In this project, We'll introduce how we used the KR260 and PYNQ-DPU for object detection, and how we controlled GPIO and PWM.

Subproject for the AMD Pervasive AI Developer Contest

This project is part of a subproject for the AMD Pervasive AI Developer Contest.

Be sure to check out the other projects as well.

***The main project is currently under submission. ***

1. PYNQ + GPIO(LED Blinking)

2. PYNQ + PWM(DC-Motor Control)

3. Object Detection(Yolo) with DPU-PYNQ

4. Implementation DPU, GPIO, and PWM

5. Remote Control 360° Camera

6. GStreamer + OpenCV with 360°Camera

7. 360 Live Streaming + Object Detect(DPU) << this project

8. ROS2 3D Marker from 360 Live Streaming

9. Control 360° Object Detection Robot Car

10. Improve Object Detection Speed with YOLOX

11. Benchmark Architectures of the DPU

12. Power Consumption of 360° Object Detection Robot Car

13. Application to Vitis AI ONNX Runtime Engine (VOE)

14. Appendix: Object Detection Using YOLOX with a Webcam

Please note that before running the above subprojects, the following setup, which is the reference for this AMDcontest, is required.

https://github.com/amd/Kria-RoboticsAI

Introduction

I conducted real-time object detection on 360 live streaming image data using the RICOH THETA 360° camera. Initially, I performed object detection (Yolo) using the DPU of KR260.

Next, I'll explain how I utilized KR260 and PYNQ-DPU for object detection and controlled LEDs (GPIO).

Additionally, I performed human object detection and automatically rotated a car robot in the direction of the detected person. This was achieved using PWM, motor driver, and DC motor.

I'll also provide explanations covering the program, GStreamer, and OpenCV processing.

GStreamer Not Included in PYNQ's OpenCV

In this project, I conducted object detection (Yolo) using PYNQ-DPU. To run the DPU, it was necessary to execute it in PYNQ's virtual environment.

Here is the check program for my environment:

https://github.com/iotengineer22/AMD-Pervasive-AI-Developer-Contest/blob/main/src/gst-dpu-test/gst-back-check.py

Below is the build information of OpenCV in PYNQ's virtual environment:

However, upon checking the build information of PYNQ's OpenCV, I found that GStreamer was not included.

(pynq-venv) root@kria:/home/ubuntu/gst-dpu-test# python3 gst-back-check.py

GStreamer:
    GStreamer:                   NO

GStreamer is essential for processing 360° video streams in real-time.

it was confirmed in the following article that GStreamer is included in the Ubuntu environment of the KR260.

6. GStreamer + OpenCV with 360°Camera << previous project

Additionally, during the Kria-PYNQ installation, opencv-python is included as indicated in the following link:

https://github.com/Xilinx/Kria-PYNQ/blob/main/install.sh

Therefore, I uninstalled opencv-python within the PYNQ virtual environment:

sudo su
source /etc/profile.d/pynq_venv.sh
pip uninstall opencv-python

After uninstalling, I switched to the Ubuntu environment on the KR260, making GStreamer available:

(pynq-venv) root@kria:/home/ubuntu/gst-dpu-test# python3 gst-back-check.py

GStreamer:
    GStreamer:                   YES (1.19.90)

360° Object Detect(DPU)

I conducted real-time object detection on 360 live streaming image data using the RICOH THETA 360° camera and DPU.

The FPGA overlay is detailed in the following project:

4. Implementation DPU, GPIO, and PWM

The test .bit,.hwh,.xclbin, and.py(app_gst-real-360-yolov3_tf2.py) files are available on GitHub.

And we have provided the .xmodel(kr260_yolov3_tf2.xmodel) at the following link. This is the model used for testing the DPU (object detection).

https://github.com/iotengineer22/AMD-Pervasive-AI-Developer-Contest/blob/main/src/gst-dpu-test/

sudo su
source /etc/profile.d/pynq_venv.sh
cd /src/gst-dpu-test/
python3 app_gst-real-360-yolov3_tf2.py

Here is test video:

Initially, the Python program splits the 2k (1920x960) 360° image data into four 480x480 images.

It is because applying a standard object detection YOLO model to a very wide 360° image makes object detection difficult.

By dividing and cropping into four 480x480 sections (or two 960x960 sections), object detection can be effectively performed in each section.

Object Detection by Splitting 360° Images

Here is a part of the program that splits 360° images and detects objects.

The image is divided, and a simple for loop is used. Please refer to the program for more details.

while True:
    ret, frame = cap.read()
    if not ret:
        break

    # Get the height and width of the image
    height, width, _ = frame.shape

    # Shift the image to the right by 240 pixels (to center the front in Section 1)
    shift = width // 8
    frame_shifted = np.roll(frame, shift, axis=1)

    # Split the shifted image into 4 sections (480:480)
    sections = [
        frame_shifted[360:840, :width // 4],
        frame_shifted[360:840, width // 4:width // 2],
        frame_shifted[360:840, width // 2:3 * width // 4],
        frame_shifted[360:840, 3 * width // 4:]
    ]

    # Display each section
    for i, section in enumerate(sections):
        run(section, i + 1, display)

        if display:
            cv2.imshow(f"Section {i+1}", section)

360° Object Detect(DPU) + GPIO

I used the KR260 and PYNQ-DPU for object detection and controlled LEDs (GPIO).

Test PCB

To verify GPIO functionality, I connected a debug (LED/SW) PCB to the PMOD connector.

The data for the debug PCB for KR260 used in this test for GPIO is available on the following GitHub:

GitHub Repository for PCB KR260 PMOD Test

The test.py (app_gst-real-360-yolov3_tf2.py) files are the same as last.

python3 app_gst-real-360-yolov3_tf2.py

Here is test video:

By detecting a ball in the 360° image, specific sections were designated to light up red, green, or blue LEDs.

As previously mentioned, the 360° image data was divided into four 480x480 images.

When a ball is detected in section 1 (front side), all LEDs turn on.

If detected in section 2 (right side), the red LED turns on.

If detected in section 3 (back side), the green LED turns on.

If detected in section 4 (left side), the blue LED turns on.

360° Object Detect(DPU) + PWM

I used the KR260 and PYNQ-DPU for object detection and controlled Motors (PWM).

Test PCB

To verify PWM functionality, I connected a Motor Driver PCB to the PMOD connector.

The data for the Motor Driver PCB for KR260 used in this test for PWM is available on the following GitHub:

GitHub Repository for PCB KR260 Motor Driver

The test.py(app_360-detect-robot-car.py) files are available on GitHub.

https://github.com/iotengineer22/AMD-Pervasive-AI-Developer-Contest/blob/main/src/gst-dpu-test/app_360-detect-robot-car.py

python3 app_360-detect-robot-car.py

Here is test video:

When a person (or their hand) is detected using a 360° camera, the motor rotates in the direction of the person.

If a person is detected at a 90° position, the motor rotates 90°.

Similarly, if a person is detected at a 180° position, the motor rotates 180°.

Test PCB

To verify PWM functionality, I connected a Motor Driver PCB to the PMOD connector.

The data for the Motor Driver PCB for KR260 used in this test for PWM is available on the following GitHub:

GitHub Repository for PCB KR260 Motor Driver

Conclusion

We performed real-time object detection on 360° live streaming image data. Using the KR260 and PYNQ-DPU, we successfully detected objects and operated GPIO and PWM.

In the next project, we plan to output the 360° object detection data to ROS2.

8. ROS2 3D Marker from 360 Live Streaming << next project