Kria Autonomous Robotic Platform (Karp) uses Vitis-AI 2.0 and Yolov4 deep neural network for real-time object detection. In this tutorial, we will see how to install Vitis-AI and Xilinx Deep Learning Processor Unit (DPU) in our Petalinux image to run Yolov4 real-time object detection network.
1. Vitis-AI and Deep Learning Processor Unit (DPU)The Vitis™ AI development environment is Xilinx’s development platform for AI inference on Xilinx hardware platforms, including both edge devices and Alveo™ cards. It consists of optimized IP, tools, libraries, models, and example designs. It is designed with high efficiency and ease of use in mind, unleashing the full potential of AI acceleration on Xilinx FPGA and ACAP.
The Vitis AI Library is a set of high-level libraries and APIs built for efficient AI inference with the Deep-Learning Processor Unit (DPU). It is built based on the Vitis AI Runtime with unified APIs and provides easy-to-use interfaces for the AI model deployment on Xilinx platforms. It is built based on the Vitis AI Runtime with unified APIs and provides easy-to-use interfaces for the AI model deployment on Xilinx platforms.
The Xilinx Deep Learning Processor Unit (DPU) is a programmable engine dedicated to convolutional neural networks. The DPU IP can be integrated as a block in the programmable logic (PL) of the selected Zynq®-7000 SoC and Zynq UltraScale™+ MPSoC devices with direct connections to the processing system (PS).
2. Installing Vitis-AI 2.0 in Petalinux imageFor installing Vitis-AI inside the Petalinux image, we need to download the corresponding meta-layer. For the Karp project, we have been using Petalinux 2021.2 version, so we need to use the rel-v2021.2 branch from the official meta-vitis-ai layer. To this layer, we have added some recipes supporting the new Vitis-AI 2.0 that can be found here. I created a fork of the repository so in order to facilitate installation so the installation is very easy.
In your Petalinux project project_spec folder you need to download the next repository:
git clone -b rel-v2021.2 https://github.com/jlamperez/meta-vitis-ai.git meta-vitis-ai
Then you need to add meta-vitis-ai to bblayers in petalinux-config->Yocto Settings → User Layers and remove the one specified by default in ${SDKBASEMETAPATH}/layers/meta-vitis-ai from bblayers.
Finally, you need to add the next recipes to your petalinux-image-minimal.bbappend.
packagegroup-petalinux-vitisai-dev \
packagegroup-petalinux-vitisai
3. Add FPGA FirmwareTo work with Vitis-Ai you need to program the Programmable Logic with a bitstream which contains the Xilinx Deep Learning Processor Unit (DPU).
In this case, I am not going to create any Vivado or Vitis design, I am going to reuse existing designs that can be found in kv260-firmware repository for my Petalinux version located in xlnx_rel_v2021.2 branch. More concretely I am going to reuse benchmark-b4096 and smartcam Programmable Logic firmware files.
These firmware files consist of:
- PL bitstream binary
- Xclbin binary
- Device tree overlay source
- json file required by dfx-mgr
Benchmark-b4096 are firmware files for benchmarking DPU networks. It consisted of a B4096 DPU IP Core. About the smartcam files and how you can use a smartcam application you can read here. The smartcam firmware uses B3136 DPU IP and adds some pre and post-processing IP Cores in the Programmable Logic.
For adding this FPGA firmware we will use fpgamanager template in Petalinux. This fpgamanager template creates application recipes that will be added to meta-user/recipes-apps.
For example, you can create benchmark-b4096.bb and smartcam.bb like this,
cd project_path
git clone -b xlnx_rel_v2021.2 https://github.com/Xilinx/kv260-firmware
And using,
# benchmark-b4096.bb
petalinux-create -t apps --template fpgamanager -n benchmark-b4096 --enable \
--srcuri "../kv260-firmware/benchmark-b4096/kv260-benchmark-b4096.bit \
kv260-firmware/benchmark-b4096/kv260-benchmark-b4096.xclbin \
kv260-firmware/benchmark-b4096/shell.json \
kv260-firmware/benchmark-b4096/kv260-benchmark-b4096.dtsi"
# smartcam.bb
petalinux-create -t apps --template fpgamanager -n karp-smartcam --enable \
--srcuri "../kv260-firmware/smartcam/kv260-smartcam.bit \
kv260-firmware/smartcam/kv260-smartcam.xclbin \
kv260-firmware/smartcam/shell.json \
kv260-firmware/smartcam/kv260-smartcam.dtsi"
Now the last thing you need to do is to configure the root file system and add the applications with,
petalinux-config -c rootfs
After all these steps you can build the image and Vitis-AI 2.0 and the firmware application will be installed inside Petalinux image.
4. Booting the image and installing the accelerator.Once you have booted your board you can see different accelerator with the next command,
xilinx-k26-starterkit-2021_2:~$ sudo xmutil listapps
Accelerator Base Type #slots Active_slot
benchmark-b4096 benchmark-b4096 XRT_FLAT 0 -1
kv260-dp kv260-dp XRT_FLAT 0 0,
karp-smartcam karp-smartcam XRT_FLAT 0 -1
Socket 9 closed by client
In this case, you can see that 3 accelerators are available inside the board, benchmark-b4096,karp-smartcam and kv260-dp. The first two are the ones created by us, the other is the kv260-dp and is used for displaying the display-port/hdmi signal to a monitor. This last accelerator is the one that is activated by default and it is marked with a 0 the other ones have a -1.
For loading our accelerators we need first to unload the display port accelerator and then load one of ours. This can be done with,
sudo xmutil unloadapp kv260-dp
sudo xmutil loadapp benchmark-b4096
After loading the accelerator you should see something like this,
xilinx-k26-starterkit-2021_2:~$ sudo xmutil loadapp benchmark-b4096
DFX-MGRD> daemon loading accel benchmark-b4096
Feb 19 03:35:10 xilinx-k26-starterkit-2021_2 kernel: [ 239.102933] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/firmware-name
Feb 19 03:35:10 xilinx-k26-starterkit-2021_2 kernel: [ 239.102945] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/resets
Feb 19 03:35:10 xilinx-k26-starterkit-2021_2 kernel: [ 239.102953] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/display@fd4a0000/status
Feb 19 03:35:10 xilinx-k26-starterkit-2021_2 kernel: [ 239.102963] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/display@fd4a0000/zynqmp_dp_snd_pcm0/status
Feb 19 03:35:10 xilinx-k26-starterkit-2021_2 kernel: [ 239.102973] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/display@fd4a0000/zynqmp_dp_snd_pcm1/status
Feb 19 03:35:10 xilinx-k26-starterkit-2021_2 kernel: [ 239.102981] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/display@fd4a0000/zynqmp_dp_snd_card/status
Feb 19 03:35:10 xilinx-k26-starterkit-2021_2 kernel: [ 239.102990] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/display@fd4a0000/zynqmp_dp_snd_codec0/status
Feb 19 03:35:10 xilinx-k26-starterkit-2021_2 kernel: [ 239.103120] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/afi0
Feb 19 03:35:10 xilinx-k26-starterkit-2021_2 kernel: [ 239.103139] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/clocking0
Feb 19 03:35:10 xilinx-k26-starterkit-2021_2 kernel: [ 239.103157] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/clocking1
Feb 19 03:35:10 xilinx-k26-starterkit-2021_2 kernel: [ 239.103175] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/zocl
Feb 19 03:35:10 xilinx-k26-starterkit-2021_2 kernel: [ 239.112406] xilinx-dp-snd-card fd4a0000.display:zynqmp_dp_snd_card: ASoC: no DMI vendor name!
Feb 19 03:35:10 xilinx-k26-starterkit-2021_2 kernel: [ 239.113000] OF: graph: no port node found in /axi/display@fd4a0000
Feb 19 03:35:10 xilinx-k26-starterkit-2021_2 kernel: [ 239.336465] zynqmp_clk_divider_set_rate() set divider failed for pl0_ref_div1, ret = -13
Feb 19 03:35:10 xilinx-k26-starterkit-2021_2 kernel: [ 239.338280] zocl-drm axi:zyxclmm_drm: IRQ index 8 not found
DFX-MGRD> Successfully loaded base design.
Accelerator loaded to slot 0
Socket 6 closed by client
And also how this is changed to 0 Active_slot.
xilinx-k26-starterkit-2021_2:~$ sudo xmutil listapps
Accelerator Base Type #slots Active_slot
benchmark-b4096 benchmark-b4096 XRT_FLAT 0 0,
kv260-dp kv260-dp XRT_FLAT 0 -1
karp-smartcam karp-smartcam XRT_FLAT 0 -1
5. Trying resnet50 example to check everything is working wellTo check that the installation has been satisfactory, we will try to launch resnet50 Vitis-AI example.
For this, we need to copy Vitis-Ai VART demo folder to our Kria board. Once done we can build the example like this,
cd /home/petalinux/VART/resnet50
./build.sh
Copy the next image to the board,
Download the resnet50 model inside the board,
wget https://www.xilinx.com/bin/public/openDownload?filename=resnet50-zcu102_zcu104_kv260-r2.0.0.tar.gz -O resnet50-zcu102_zcu104_kv260-r2.0.0.tar.gz
sudo mkdir -p /usr/share/vitis_ai_library/models
tar -xzvf resnet50-zcu102_zcu104_kv260-r2.0.0.tar.gz
sudo cp resnet50 /usr/share/vitis_ai_library/models -r
And launch the example:
xilinx-k26-starterkit-2021_2:~/VART/resnet50$ ./resnet50 /usr/share/vitis_ai_library/models/resnet50/resnet50.xmodel
If everything is okay you should see this output,
xilinx-k26-starterkit-2021_2:~/VART/resnet50$ ./resnet50 /usr/share/vitis_ai_library/models/resnet50/resnet50.xmodel
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0219 05:17:02.825711 1709 main.cc:292] create running for subgraph: subgraph_conv1
Feb 19 05:17:03 xilinx-k26-starterkit-2021_2 kernel: [ 2126.720076] [drm] ERT_EXEC_WRITE is obsoleted, use ERT_START_KEY_VAL
Image : 001.jpg
top[0] prob = 0.982662 name = brain coral
top[1] prob = 0.008502 name = coral reef
top[2] prob = 0.006621 name = jackfruit, jak, jack
top[3] prob = 0.000543 name = puffer, pufferfish, blowfish, globefish
top[4] prob = 0.000330 name = eel
6. DRM and HDMI output.Kria Autonomous Robotic Platform uses an 11.6inch HDMI LCD Monitor so we can visualize how different Vitis-AI networks work. We have launched/reused some Vitis-AI examples like multitask_v3_quad_windows or created our own Multi Yolov4 example for Karp. These examples use the Xilinx DRM driver for displaying neural network images in the Monitor.
7. multitask_v3_quad_windowsMultiTask V3 aims to do different tasks in autonomous driving scenarios simultaneously while achieving good performance and efficiency. The tasks include object detection, segmentation, lane detection, drivable area segmentation, and depth estimation, which are important components of the autonomous driving perception module.
This example can be easily run if you copy the Vitis-AI-Library to the board, build the application, download the models and run it.
cd Vitis-AI-Library/apps/multitask_v3_quad_windows
chmod +x build*
./builddrm.sh
wget https://www.xilinx.com/bin/public/openDownload?filename=multi_task_v3_pt-zcu102_zcu104_kv260-r2.0.0.tar.gz -O multi_task_v3_pt-zcu102_zcu104_kv260-r2.0.0.tar.gz
tar xvzf multi_task_v3_pt-zcu102_zcu104_kv260-r2.0.0.tar.gz
sudo cp -r multi_task_v3_pt /usr/share/vitis_ai_library/models/
sudo ./multitaskv3_quad_windows_drm d58cbda2-97976be7__640x360.avi -t 4 > /dev/null 2>&1
You should see something like this,
This is a really good example to run in our Karp robot.
A video of the network running while Karp moves can be found here,
8. Multi-Yolov4 exampleKarp robot uses two cameras (one in front and the other one at the back) to see the environment and the Yolov4 network uses this camera input for detecting different objects in the scene.
For this example, we have used the two different Yolov4 models that are available in the Xilinx model zoo and downloaded them to the board, yolov4_leaky_spp_m, and yolov4_leaky_spp_m_pruned_0_36.
Thiscan be done like this,
wget https://www.xilinx.com/bin/public/openDownload?filename=yolov4_leaky_spp_m-zcu102_zcu104_kv260-r2.0.0.tar.gz -O yolov4_leaky_spp_m-zcu102_zcu104_kv260-r2.0.0.tar.gz
tar xvzf yolov4_leaky_spp_m-zcu102_zcu104_kv260-r2.0.0.tar.gz
sudo cp -r yolov4_leaky_spp_m /usr/share/vitis_ai_library/models/
wget https://www.xilinx.com/bin/public/openDownload?filename=yolov4_leaky_spp_m_pruned_0_36-zcu102_zcu104_kv260-r2.0.0.tar.gz -O yolov4_leaky_spp_m_pruned_0_36-zcu102_zcu104_kv260-r2.0.0.tar.gz
tar xvzf yolov4_leaky_spp_m_pruned_0_36-zcu102_zcu104_kv260-r2.0.0.tar.gz
sudo cp -r yolov4_leaky_spp_m_pruned_0_36 /usr/share/vitis_ai_library/models/
The code for running this application is available in the Code section.
Once the application is compiled with the builddrm.sh script you can launch it like this,
sudo ./karp_yolov4_drm 0 2 -t 3 -t 3 > /dev/null 2>&1
Where 0 and 2 are Petalinux video camera devices /dev/video0 and /dev/video2.
Some videos of the application working,
Comments
Please log in or sign up to comment.