ROS2 3D Marker from 360 Live Streaming with KR260

We experimented with processing 360° camera images using ROS2 Rviz2.

We managed real-time object detection from 360° live streaming data.

In this project, we'll introduce the program and share test videos.

Subproject for the AMD Pervasive AI Developer Contest

This project is part of a subproject for the AMD Pervasive AI Developer Contest.

Be sure to check out the other projects as well.

***The main project is currently under submission. ***

0. Main project << under submission

1. PYNQ + GPIO(LED Blinking)

2. PYNQ + PWM(DC-Motor Control)

3. Object Detection(Yolo) with DPU-PYNQ

4. Implementation DPU, GPIO, and PWM

5. Remote Control 360° Camera

6. GStreamer + OpenCV with 360°Camera

7. 360 Live Streaming + Object Detect(DPU)

8. ROS2 3D Marker from 360 Live Streaming << this project

9. Control 360° Object Detection Robot Car

10. Improve Object Detection Speed with YOLOX

11. Benchmark Architectures of the DPU

12. Power Consumption of 360° Object Detection Robot Car

13. Application to Vitis AI ONNX Runtime Engine (VOE)

14. Appendix: Object Detection Using YOLOX with a Webcam

Please note that before running the above subprojects, the following setup, which is the reference for this AMDcontest, is required.

https://github.com/amd/Kria-RoboticsAI

Introduction

Using ROS2's Rviz2 and KR260, we processed 360° camera images.

We used real-time object detection data from 360° live streaming.

We placed the detected objects' bounding boxes as markers in Rviz2.

Test Videos

Here is one of the actual test videos below.

We are outputting markers in ROS2 for objects detected in all directions from a 360° camera.

We'll introduce the program and share the test videos.

Installing ROS2 Rviz2

Rviz2 is a 3D visualization tool for ROS2 (Robot Operating System 2).

This time, we'll display object detection data from 360° live streaming images in Rviz2.

Install it with the following steps:

sudo apt update
sudo apt install ros-humble-rviz2
sudo su
source /opt/ros/humble/setup.bash 
rviz2

Installing OpenCV Related Packages for ROS2

We are acquiring and processing images from the 360° camera via OpenCV.

To use it with ROS2, we'll also install related libraries.

sudo apt install ros-humble-image-transport
sudo apt install ros-humble-cv-bridge

Operating Rviz2

Start Rviz2 with the following command:

sudo su
source /opt/ros/humble/setup.bash
rviz2

To display images published in Rviz2:

• In the left "Displays" panel, select "Add".
• Choose "Image".
• Select the "Topic" and enter the published image topic (e.g., /camera/image).

To display markers published in Rviz2:

• In the left "Displays" panel, find the "Global Options" settings.
• Change the "Fixed Frame" setting to match the published frame (e.g., "base_link"). This ensures it matches the frame ID used by the markers.
• Select "Add" and choose "MarkerArray" to visualize the markers in the central view window.

Publishing Markers in ROS2

We are publishing markers to ROS2.

The following example is a part of the Python code that splits a 360° image into four sections. Please refer to the program for more details.

(Note: The sizes and positions of the markers are placed rather arbitrarily.)

def publish_markers(publisher, node, detections, classes, section):
    marker_array = MarkerArray()

    if section == 'q1':  # front
        m_offset = [0, 1]
        theta = 0
        id_offset = 0
    elif section == 'q2':  # right
        m_offset = [1, 0]
        theta = 270
        id_offset = 10
    elif section == 'q3':  # back
        m_offset = [0, -1]
        theta = 180
        id_offset = 20
    elif section == 'q4':  # left
        m_offset = [-1, 0]
        theta = 90
        id_offset = 30

    for i, det in enumerate(detections):
        marker = Marker()
        marker.header.frame_id = "base_link"
        marker.header.stamp = node.get_clock().now().to_msg()
        marker.ns = "yolo_boxes"
        marker.id = i + id_offset
        marker.action = Marker.ADD
        if classes[i] in (32, 51):  # Sports Ball or Orange
            marker.type = Marker.SPHERE
        else:
            marker.type = Marker.CUBE

        p_adj = 20
        s_adj = 10
        tmp_marker_x = (det[0] + det[2]) / (2 * p_adj)
        tmp_marker_y = (det[1] + det[3]) / (2 * p_adj)
        x_rotated, y_rotated = rotate_coordinates(tmp_marker_x, tmp_marker_y, theta)
        marker.pose.position.x = x_rotated + m_offset[0]
        marker.pose.position.y = y_rotated + m_offset[1]
        marker.pose.position.z = 0.0
        marker.pose.orientation.w = 1.0
        marker.scale.x = float(det[2] - det[0]) / s_adj
        marker.scale.y = float(det[3] - det[1]) / s_adj
        marker.scale.z = float(det[3] - det[1]) / s_adj  # Height of the box

        det_color = colors[classes[i]]
        marker.color.a = 0.5  # Transparency
        marker.color.r = float(det_color[2])
        marker.color.g = float(det_color[1])
        marker.color.b = float(det_color[0])
        marker.lifetime = Duration(seconds=2).to_msg()
        marker_array.markers.append(marker)
    
    publisher.publish(marker_array)

ROS2 3D Marker from 360 Live Streaming Test1

I conducted real-time object detection on 360 live streaming image data using the RICOH THETA 360° camera and DPU.

The FPGA overlay is detailed in the following project:

4. Implementation DPU, GPIO, and PWM

Although it's not related to the current subproject, we will go ahead and install the game controller library below.

sudo su
source /etc/profile.d/pynq_venv.sh
pip install inputs

The test .bit,.hwh,.xclbin, and.py(gst-ros2-360-detect-car.py) files are available on GitHub.

And we have provided the .xmodel(kr260_yolov3_tf2.xmodel) at the following link. This is the model used for testing the DPU (object detection).

https://github.com/iotengineer22/AMD-Pervasive-AI-Developer-Contest/tree/main/src/gst-ros2

sudo su
source /etc/profile.d/pynq_venv.sh
source /opt/ros/humble/setup.bash
cd /src/gst-ros2/
python3 gst-ros2-360-detect-car.py

Here is test video:

We display images and markers published from KR260 using ROS2's Rviz2.

The KR260 splits a 360° image into four sections and performs object detection and marker output in each section.

We are conducting tests to pick up and transport balls around the 360° camera. The real-time changes, including the ROS2 markers, can be observed.

ROS2 3D Marker from 360 Live Streaming Test2

The next test uses the below program.

python3 gst-ros2-360-2divide.py

But this time we will differentiate detected objects and change the marker output accordingly.

• Sports ball or orange: sphere (Marker.SPHERE)
• Others (e.g., PET bottle): cube (Marker.CUBE)

A portion of the program looks like this:

if classes[i] in (32, 51):  # Sports Ball or Orange
    marker.type = Marker.SPHERE
else:
    marker.type = Marker.CUBE
# marker.type = Marker.SPHERE

Here is test video:

In this test, the KR260 splits the 360° image into two sections, front and back.

The program outputs markers as cubes for PET bottles and as spheres for sports balls.

During the test, when objects are moved, we can see the marker outputs changing in real-time.

Summary

We processed 360° camera images using ROS2's Rviz2.

We were able to output ROS2 markers in real-time from 360° live streaming data.

In the next project, we will introduce the operation of 360° object detection robotic car.

9. Control 360° Object Detection Robot Car << next project