Hardware components | ||||||
 |
| × | 1 | |||
| × | 1 | ||||
Software apps and online services | ||||||
 |
| |||||
 |
| |||||
 |
| |||||
We tried object detection with a regular USB-connected webcam using the KR260.
By using the FPGA’s DPU for YOLOX inference, we can achieve fast, real-time detection.
Subproject for the AMD Pervasive AI Developer ContestThis project is part of a subproject for the AMD Pervasive AI Developer Contest.
Be sure to check out the other projects as well.
**The main project is currently under submission. ***
0. Main project << under submission
2. PYNQ + PWM(DC-Motor Control)
3. Object Detection(Yolo) with DPU-PYNQ
4. Implementation DPU, GPIO, and PWM
6. GStreamer + OpenCV with 360°Camera
7. 360 Live Streaming + Object Detect(DPU)
8. ROS2 3D Marker from 360 Live Streaming
9. Control 360° Object Detection Robot Car
10. Improve Object Detection Speed with YOLOX
11. Benchmark Architectures of the DPU
12. Power Consumption of 360° Object Detection Robot Car
13. Application to Vitis AI ONNX Runtime Engine (VOE)
14. Appendix: Object Detection Using YOLOX with a Webcam << this project
Please note that before running the above subprojects, the following setup, which is the reference for this AMDcontest, is required.
https://github.com/amd/Kria-RoboticsAI
IntroductionMany people may not have the 360° camera required for Main project.
Therefore, as a reference, we will introduce a subproject using a generic webcam, as well as controlling DPU, GPIO and Output ROS2.
We tried object detection with a regular USB-connected webcam using the KR260.
By using the FPGA’s DPU for YOLOX inference, we can achieve fast, real-time detection.
We will introduce the program along with the test results.
Object Detection with DPU and YOLOX from a USB WebcamI tried object detection with a regular USB-connected webcam using the KR260.
By using the FPGA’s DPU for YOLOX inference, we can achieve fast, real-time detection.
Here is the test video:
As for the normal speed of the webcam, 640x480 at 25 fps can be confirmed.
Even with YOLOX, a real-time speed of 17 fps was achieved.
(The GPIO is also operated simultaneously with object detection.)
Because we use the DPU instead of the CPU, we can perform high-speed object detection.
We will introduce the program along with the test results.
Install libraries to enable live streaming from a USB camera.
GStreamer, libuvc, and v4l2loopback-dkms are required.
The installation method is introduced in the following subproject:
6. GStreamer + OpenCV with 360°Camera
We also provide an environment to use GStreamer with the PYNQ environment.
(This is because we drive a check LED from GPIO using PYNQ.)
This is introduced in the following subproject:
7. 360 Live Streaming + Object Detect(DPU)
Webcam in Use: logicool C270nThe webcam used is the "logicool C270n."
It is a very affordable webcam, but (according to specifications) it has a speed of 640x480 at 30 fps. It is sufficient for testing purposes.
When you connect the webcam to the KR260, you can verify the device.
ubuntu@kria:~$ sudo dmesg | grep usb
[ 8.850427] input: UVC Camera (046d:0825) as /devices/platform/axi/ff9d0000.usb/fe200000.usb/xhci-hcd.1.auto/usb1/1-1/1-1.2/1-1.2:1.0/input/input1Using v4l2-ctl, you can verify that it is recognized as video0.
root@kria:/home/ubuntu# v4l2-ctl --list-device
UVC Camera (046d:0825) (usb-xhci-hcd.1.auto-1.2):
/dev/video0
/dev/video1
/dev/media0You can also check which formats the camera supports.
Here is the actual information from the webcam:
root@kria:/home/ubuntu# v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture
[0]: 'YUYV' (YUYV 4:2:2)
Size: Discrete 640x480
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)The program is saved in the following GitHub repository:
https://github.com/iotengineer22/AMD-Pervasive-AI-Developer-Contest/tree/main/src/usb-camera
The live streaming is done via USB with the following pipeline:
It is set to 640x480 at 30 fps.
# Definition of the GStreamer pipeline (software)
pipeline = "v4l2src device=/dev/video0 ! video/x-raw, width=640, height=480, framerate=30/1 ! videoconvert ! appsink"From here, it is the same as the demo of the test video.
First, test the program that only does live streaming (usb-camera-test.py).
Live streaming on the KR260 was successful.
sudo su
cd /home/ubuntu/AMD-Pervasive-AI-Developer-Contest/src/usb-camera/
source /etc/profile.d/pynq_venv.sh
python3 usb-camera-test.pyWhen counting on the program, it was actually about 25 fps.
Perform object detection using live streaming from the webcam.
We use the KR260’s DPU and YOLOX (app_gst-yolox-real-normal-camera-gpio.py).
This time, when a yellow ball (sports ball) is detected, the LED (GPIO) is turned on.
python3 app_gst-yolox-real-normal-camera-gpio.pyThe GPIO is controlled from the PL (FPGA) IP.
The DPU and FPGA files (dpu.bit, dpu.hwh, dpu.xclbin) are also saved in the same repository.
https://github.com/iotengineer22/AMD-Pervasive-AI-Developer-Contest/tree/main/src/usb-camera
For details on the integration method, including DPU and GPIO, please refer to the following subproject:
4. Implementation DPU, GPIO, and PWM
When the program starts, you can see the webcam image with added object detection.
In the actual live streaming with YOLOX, about 17 fps is achieved.
Since it would be only a few fps with a CPU, you can see that the DPU is properly accelerating it.
A comparison between CPU and DPU is introduced in the following subproject:
13. Application to Vitis AI ONNX Runtime Engine (VOE)
When a yellow ball (sports ball) is detected, you can see that the LED (GPIO) is turned on.
This is operated from GPIO via the PMOD connector.
We will test outputting Marker and Image data to ROS2 using data from a webcam.
Please refer to the following subproject for installing the libraries required for ROS2.
8. ROS2 3D Marker from 360 Live Streaming
To visualize ROS2, start rviz2.
sudo su
source /opt/ros/humble/setup.bash
rviz2Once rviz2 is set up, start the program (gst-yolox-ros2-normal-camera.py).
sudo su
source /etc/profile.d/pynq_venv.sh
source /opt/ros/humble/setup.bash
cd /home/ubuntu/AMD-Pervasive-AI-Developer-Contest/src/usb-camera/
python3 gst-yolox-ros2-normal-camera.pyThe test video is as follows:
Using the information detected by YOLOX from the webcam, the output to ROS2 is executed at about 17 fps.
Markers and images are published to ROS2 without any issues.
Many people may not have the 360° camera required for Main project.
Therefore, as a reference, we will introduce a subproject using a generic webcam, as well as controlling DPU, GPIO and Output ROS2.
In the next project, we will introduce main project for the AMD Pervasive AI Developer Contest.
0. Main project << under submission
Comments
Please log in or sign up to comment.