Hardware components | ||||||
 |
| × | 1 | |||
 |
| × | 1 | |||
Software apps and online services | ||||||
 |
| |||||
In this project, we'll walk you through how we achieved real-time image processing using a KR260 and a 360° camera (RICOH THETA).
We'll cover the installation methods and the programs used.
This project is part of a subproject for the AMD Pervasive AI Developer Contest.
Be sure to check out the other projects as well.
***The main project is currently under submission. ***
0. Main project << under submission
2. PYNQ + PWM(DC-Motor Control)
3. Object Detection(Yolo) with DPU-PYNQ
4. Implementation DPU, GPIO, and PWM
6. GStreamer + OpenCV with 360° Camera << this project
7. 360 Live Streaming + Object Detect(DPU)
8. ROS2 3D Marker from 360 Live Streaming
9 Control 360° Object Detection Robot Car
10. Improve Object Detection Speed with YOLOX
11. Benchmark Architectures of the DPU
12. Power Consumption of 360° Object Detection Robot Car
13. Application to Vitis AI ONNX Runtime Engine (VOE)
14. Appendix: Object Detection Using YOLOX with a Webcam
Please note that before running the above subprojects, the following setup, which is the reference for this AMDcontest, is required.
https://github.com/amd/Kria-RoboticsAI
IntroductionWe connected the pipeline using GStreamer and processed it with OpenCV.
This setup allows us to obtain 360° live streaming data via USB from KR260.
Testing with Real-Time 360° Image
We tested the setup by throwing a ball around the 360° camera.
The following image was captured in real-time from the RICOH THETA on the KR260. Viewing the image from above provides a clear perspective.
360° Image Splitting and Shifting with OpenCV
We further processed the image using OpenCV to split the 360° image into four parts and shift the screen position.
This was practice for preprocessing (pre_process) object detection using DPU.
Test Video
Below are 360° live streaming test videos:
Installation Methods and Programslibuvc
We use libuvc to obtain video streams from the USB camera (RICOH THETA).
You need to download and install the necessary UVC (USB Video Class) libraries.
Refer to the following GitHub repository:
https://github.com/nickel110/libuvc.git
sudo apt update
sudo apt install cmake
sudo apt install libusb-1.0-0-dev
sudo su
git clone https://github.com/nickel110/libuvc.git
cd libuvc/
mkdir build
cd build/
cmake ..
make && sudo make installGStreamer
We use Gstreamer pipelines to encode the video stream.
Refer to the official GStreamer documentation:
https://gstreamer.freedesktop.org/documentation/installing/on-linux.html?gi-language=c
Install GStreamer with the following command:
sudo apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudiov4l2loopback-dkms
Installing v4l2loopback-dkms creates virtual video devices.
sudo apt install v4l2loopback-dkmsgstthetauvc
Download and install the Gstreamer THETA UVC plugin (gstthetauvc) for RICOH THETA.
Refer to the following GitHub repository:
https://github.com/nickel110/gstthetauvc.git
git clone https://github.com/nickel110/gstthetauvc.git
cd gstthetauvc/thetauvc/
makeMove the created gstthetauvc.so file to the GStreamer plugins folder. Locate gstreamer-1.0 and copy the file:
sudo find / -type d -name 'gstreamer-1.0'
ls /usr/lib/aarch64-linux-gnu/gstreamer-1.0
sudo cp gstthetauvc.so /usr/lib/aarch64-linux-gnu/gstreamer-1.0
ls /usr/lib/aarch64-linux-gnu/gstreamer-1.0Update the library links and cache:
sudo /sbin/ldconfig -vCheck if the gstthetauvc plugin is available:
gst-inspect-1.0 thetauvcsrcFor the tests, we used Ubuntu's GUI via the KR260's display port.
In this project, we ran the tests in a regular root environment rather than the PYNQ virtual environment.
Check the OpenCV build information and ensure it supports GStreamer.
The test program is available on GitHub.
Running the program confirms the OpenCV build information, and the demo screen is displayed via GStreamer.
The confirmation message "GStreamer: YES (1.19.90)" should be visible.
root@kria:/home/ubuntu# python3 gst-back-check.py
GStreamer: GStreamer: YES (1.19.90)I'll demonstrate how to perform image processing using 360° camera(RICOH THETA) with GStreamer and OpenCV.
We will process 2K (1920x960) 360° video streams using GStreamer.
GStreamer Pipeline
This time, the pipeline used to obtain 360° live streaming from THETA on the KR260 is as follows.
After defining the pipeline, it's passed to OpenCV.
Here's an excerpt of the program:
# Definition of the GStreamer pipeline (software)
pipeline = "thetauvcsrc mode=2K ! h264parse ! avdec_h264 ! videoconvert ! video/x-raw,format=BGR ! appsink"
# Initialize the VideoCapture object
cap = cv2.VideoCapture(pipeline, cv2.CAP_GSTREAMER)
while True:
ret, frame = cap.read()
if not ret:
print("Failed to receive the frame.")
break
# Display the full frame
cv2.imshow("Full Frame", frame)First, we capture 360° images directly from the RICOH THETA on the KR260.
This setup allows for real-time 360° live streaming on the screen.
The program is written in Python and is available on GitHub.
We tested the setup by throwing a ball around the 360° camera.
python3 gst-test-360-no-divide.pyThe test video is below:
The KR260 captures the 360° video stream from the RICOH THETA in real-time.
The pipeline configuration uses OpenCV only for display purposes with imshow. We capture and display 2K (1920x960) image data.
We tested by throwing a ball around the 360° camera. Viewing the image from above provides a clear perspective.
Additionally, we practiced pre-processing (pre_process) for object detection on DPU by using OpenCV to split the 360° image into four sections and shift the screen position.
The program is written in Python and is available on GitHub.
python3 gst-test-360-4divide.pyThe test video is below:
We divided the 1920x960 image into four sections of 480x960 each.
We also shifted the image so that the front of the 360° camera is centered.
In this project, we used the software decoder (avdec_h264) in the GStreamer pipeline.
However, the KR260’s Zynq UltraScale+ EV includes H.264/H.265 video codecs.
Our original intention was to use the hardware H.264/H.265 encoder/decoder.
Upon checking the plugins available on KR260’s Ubuntu, we found no hardware encoders/decoders.
root@kria:/home/ubuntu# gst-inspect-1.0 | grep h264
libav: avdec_h264: libav H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 decoder
libav: avenc_h264_omx: libav OpenMAX IL H.264 video encoder encoder
openh264: openh264dec: OpenH264 video decoder
openh264: openh264enc: OpenH264 video encoder
rtp: rtph264depay: RTP H264 depayloader
rtp: rtph264pay: RTP H264 payloader
typefindfunctions: video/x-h264: h264, x264, 264
uvch264: uvch264deviceprovider (GstDeviceProviderFactory)
uvch264: uvch264mjpgdemux: UVC H264 MJPG Demuxer
uvch264: uvch264src: UVC H264 Source
videoparsersbad: h264parse: H.264 parserAccording to the official documentation, it needs to be included with Petalinux:
https://docs.amd.com/r/en-US/ug1449-multimedia/GStreamer-Multimedia-Framework
We verified the official plugin method, but no hardware encoders/decoders were available. Although we will not use hardware encoders/decoders this time, we aim to try them in the future.
ReferenceMany thanks for reference articles and repositories.
https://github.com/nickel110/libuvc.git
https://gstreamer.freedesktop.org/documentation/installing/on-linux.html?gi-language=c
https://github.com/nickel110/gstthetauvc.git
SummaryWe achieved real-time image processing with the KR260 and a 360° camera (RICOH THETA) by connecting the pipeline with GStreamer and processing with OpenCV.
In the next post, we will perform real-time 360° object detection.
7. 360 Live Streaming + Object Detect(DPU)<< next project
Comments