Super-Resolution with Vitis-AI on ZCU104 custom platform

Introduction

The Vitis AI accelerates AI inference on Xilinx hardware platforms, including both Edge devices and Alveo accelerator cards. It makes it easy for users without FPGA knowledge to develop deep-learning inference applications.

This project demonstrates how to create a custom Vitis unified software platform for the Xilinx Zynq MPSoC ZCU104 evaluation board and run a Super-Resolution AI application on the created platform. Keep in mind that even though the target board of the project is the ZCU104 evaluation board, you can apply the same flow to other embedded platforms with small changes.

Let's do this!

Step 1: Creating the Vitis Platform

One way of using Vitis-AI on Xilinx embedded platforms is using a pre-built SD Card Vitis-AI Image which is normally provided by Board Vendors. As of the time of this project, the latest ZCU104 Vitis-AI SD Card Image can be found here. This is a quick and easy way to start but using the pre-built image limits the ability to change any thing in the Hardware or Software side of the platform. So instead we are going to create a custom platform.

This way we can:

• Customize hardware components in the Hardware platform in VIVADO to be implemented on the PL. (such as VCU, MIG, PLL,...)
• Customize the embedded Linux image in petalinux and add/remove any packages/software components we require in our application.
• Customize the DPU Kernels and change the configurations explained in the DPU TRD if needed.
• Add/Remove other Hardware Kernels in the Xilinx Vitis flow beside DPU kernels to be used in the application.

To do this, we are going to use the Vitis platform creation tutorial on ZCU104 of the Xilinx Vitis-Tutorials git. We'll call the workspace directory on our host machine $workspace, vitis installation directory $vitis_dir, petalinux installation directory $petalinux_dir and the platform creation tutorial "the tutorial" from now on.

$ cd $workspace
$ git clone https://github.com/Xilinx/Vitis-Tutorials.git
$ cd Vitis-Tutorials/

We will be using the 2021.1 branch of the git which is the default branch at the time of this project. The platform created from this branch was tested on Vitis 2021.2 and Vitis AI 1.4.1. Use the command below to go to this branch if you're not already on it.

$ git checkout 2021.1 
$ cd Vitis_Platform_Creation/Introduction/02-Edge-AI-ZCU104/ref_files

Now we need to run the Makefile script for creating the platform for ZCU104. I recommend reading all the steps of the tutorial carefully, (this way you can customize the HW/SW part later if needed) But using the Makefile script provided by Xilinx to build the platform instead of doing the steps manually. Don't forget to source Xilinx tool sets before running the script.

$ source $vitis_dir/Vitis/2021.2/settings64.sh
$ source $petalinux_dir/2021.2/settings.sh
$ make all

Note: The "petalinux-build" or "petalinux-build --sdk" command in step2 of the script may fail on the first try, causing the script to stop. If that happened you need to re-run that step and run the next steps yourself:

$ make -C step2_petalinux petalinux_build
$ make -C step2_petalinux petalinux_build_sdk
$ make -C step3_pfm all

After the script is finished your directory should look something like below. Important directories are explained in the tree.

├── step1_vivado
│   └── build
│       └── vivado
│           └── myproj : VIVADO project for Hardware platform
├── step2_petalinux
│   └── build
│       └── petalinux : Petalinux project for Software platform ($peta_prj)
│           ├── build
│           ├── components
│           ├── images
│           └── project-spec
├── step3_pfm
│   ├── boot : linux image components (copied from $peta_prj/images/linux)
│   ├── platform_repo
│   │   ├── Packages
│   │   └── zcu104_custom_platform : custom Vitis Platform ($platform_dir)
│   │       ├── export
│   │       └── hw
│   ├── sd_dir : linux boot components (copied from $peta_prj/images/linux)
│   └── sw_comp
│       └── sysroots : sysroot for cross-compiling apps for Arm cores
│           ├── cortexa72-cortexa53-xilinx-linux
│           └── x86_64-petalinux-linux
└── step4_validate

Now that the Vitis platform is created, you can customize each part for your later applications.

- Optional But Recommended:

Note that the Makefile script skips some optional steps described in the tutorial (such as part 3 and 5 of this section). Some of these steps help in running the Vitis AI application in the next sections so let's do them manually.

1. Adding some packages to rootfs for running Vitis AI application on target board:

$ cd step2_petalinux/build/petalinux/
$ petalinux-config -c rootfs

In the pop-up menu, make sure the following packages are selected:

In Petalinux Package Groups -> 
packagegroup-petalinux-gstreamer -> packagegroup-petalinux-gstreamer
packagegroup-petalinux-opencv -> packagegroup-petalinux-opencv
packagegroup-petalinux-opencv -> packagegroup-petalinux-opencv-dev
packagegroup-petalinux-self-hosted -> packagegroup-petalinux-self-hosted

Exit the menu and save.

2. Doing the optional steps of the tutorial:

Do the "step 3: Enable OpenSSH and disable dropbear" and "step 5: Disable CPU IDLE in kernel config" described in this section.

Finally build the Linux image again and copy the files to the corresponding directories.

$ petalinux-build
$ cd ../../../step3_pfm/
$ cp ../step2_petalinux/build/petalinux/images/linux/zynqmp_fsbl.elf ./boot/
$ cp ../step2_petalinux/build/petalinux/images/linux/pmufw.elf ./boot/
$ cp ../step2_petalinux/build/petalinux/images/linux/bl31.elf ./boot/
$ cp ../step2_petalinux/build/petalinux/images/linux/u-boot-dtb.elf ./boot/u-boot.elf
$ cp ../step2_petalinux/build/petalinux/images/linux/system.dtb ./boot/
$ cp ../step2_petalinux/build/petalinux/images/linux/boot.scr ./sd_dir/
$ cp ../step2_petalinux/build/petalinux/images/linux/system.dtb ./sd_dir/
$ cp ../step2_petalinux/build/petalinux/images/linux/rootfs.ext4 ./sw_comp
$ cp ../step2_petalinux/build/petalinux/images/linux/Image ./sw_comp

Step 2: Creating the Platform project in the Vitis IDE

Now that the platform files are created, we can create and build the platform project in the IDE.

Open vitis.

$ vitis &

select the zcu104_custom_platform ($platform_dir) folder as workspace.

$workspace/Vitis-Tutorials/Vitis_Platform_Creation/..../ref_files/step3_pfm/platform_repo/zcu104_custom_platform

Create a new platform project. In the "Platform project name" section insert zcu104_base. In the next windows go to "select a platform from repository" tab and click on Add, then select the $platform_dir/export directory. You should see the custom platform added to the repo. Select it and select finish. Finally right-click on the zcu104_base project in the explorer window and select Build project to build the platform project.

Note that the reason for doing this step is that the currently Vitis AI TRD application project that we'll create in the next step will look at the platform name and search for the keyword "zcu104_base" in it. Therefore if the platform name is "zcu104_custom_platform" as the Makefile script has created, some project configurations will not load automatically. This issue is described in issue#122.

Step 3: Creating the Vitis-AI Template Application project

To create the Vitis-AI application in Vitis IDE, we'll use the DPU TRD template application project in the Vitis-AI repo.

Do the steps 1 to 8 of the Test 3 of step 4 of the tutorial. (We will modify the last steps to add our custom application (SR) to the project.)

Note1: Don't forget to select the "zcu104_base" platform project we created in the previous section instead of "zcu104_custom_platform" in step 4.

Note2: For the rootfs ans Kernel Image locations, use $peta_prj/images/linux/rootfs.ext4 and $peta_prj/images/linux/Image files.

Note3: If in the second step, Vitis failed to download the Vitis-AI library, you can clone the library outside of Vitis and point to the cloned directory in the "Location" section. After that there should be no need to download the repo because we've already cloned it.

$ cd $workspace
$ git clone https://github.com/Xilinx/Vitis-AI.git

After completing all 8 steps, your application project should have 3 sub-projects. One for HW kernels (DPUs), one for HW link (VIVADO linking kernels to Hardware platform) and one for the Host (PS) application. You can customize DPU kernel configurations such as number of DPU cores and other configurations described in the DPU TRD for your later application specific requirements. Diagram of the system we just created looks like this:

Remove the Host application by right-clicking on it and selecting Delete. We'll create our own host application (Super-Resolution) in the next step.

Step 4: Creating the Super-Resolution application

• From the menu select File -> New -> Application Project.
• Select the zcu104_base platform we created previously and click next.
• In the Application project name type rcan
• Under "select a system project" select dpu_trd_system. This way the host application project we'll be created as a part of the DPU template project we created in Step 3.
• Click next twice.
• Select "Empty application" for the project template and click Finish.
• Under the created project, right-click on "src" and select "Import Sources..."
• From $workspace/Vitis-AI/demo/Vitis-AI-Library/samples/rcan/ directory, import "test_jpeg_rcan.cpp" and "process_result.hpp" files.
• Create a new folder and name it "data" inside rcan project and add your image input data to it. Input image files used in the project are shown below:

• Download the pre-built "rcan_pruned_tf" model from the Vitis-AI model zoo and copy the files into the data folder:

$ cd $workspace/Vitis-AI/models/AI-Model-Zoo/model-list/tf_rcan_DIV2K_360_640_0.98_86.95G_1.4/
$ cat model.yaml
.....
- name: tf_rcan_DIV2K_360_640_0.98_86.95G_1.4
  type: float & quantized
  board: GPU
  download link: https://www.xilinx.com/bin/public/openDownload?filename=tf_rcan_DIV2K_360_640_0.98_86.95G_1.4.zip
  checksum: 7d8569027add120a185f2dd454459135
- name: rcan_pruned_tf
  type: xmodel
  board: zcu102 & zcu104 & kv260
  download link: https://www.xilinx.com/bin/public/openDownload?filename=rcan_pruned_tf-zcu102_zcu104_kv260-r1.4.0.tar.gz
  checksum: 4c32079995fae6b22f1b78af1ce3514b
.......

$ wget https://www.xilinx.com/bin/public/openDownload?filename=rcan_pruned_tf-zcu102_zcu104_kv260-r1.4.0.tar.gz -O rcan_pruned_tf-zcu102_zcu104_kv260-r1.4.0.tar.gz
$ tar -xzvf rcan_pruned_tf-zcu102_zcu104_kv260-r1.4.0.tar.gz
$ cp -r rcan_pruned_tf $platform_dir/rcan/data

• Download Vitis AI runtime package for Edge and copy it to the data folder:

$ wget https://www.xilinx.com/bin/public/openDownload?filename=vitis-ai-runtime-1.4.0.tar.gz -O vitis-ai-runtime-1.4.0.tar.gz
$ tar -xzvf vitis-ai-runtime-1.4.0.tar.gz
$ cp -r vitis-ai-runtime-1.4.0 $platform_dir/rcan/data

• copy the dpu_sw_optimize.tar.gz file from DPU TRD to the data folder:

$ cd $workspace/Vitis-AI/dsa/DPU-TRD/app
$ cp dpu_sw_optimize.tar.gz %platform_dir/rcan/data

Now we need to set the compiler settings for the application.

• Right-click on the rcan project and select C/C++ Build Settings.
• Under "GCC Host Compiler -> Includes" add ${SYSROOT}/usr/include/opencv4
• Under "GCC Host Compiler -> Miscellaneous" append -std=c++17 to "Other flags"
• Under "GCC Host Linker -> Libraries" add the libraries needed to compile the application code to the -l options.. These libraries can be found in the build.sh file located in $workspace/Vitis-AI/demo/Vitis-AI-Library/samples/rcan/ directory. The libraries should look like this:

• Select Apply and close.
• In the explorer window, select the "dpu_trd_system.sprj" system project settings file and add --package.sd_dir=../../rcan/data to the Packaging options. This way the data folder will be packaged into the SD card content.
• Finally right-click on the "dpu_trd_system" system project and select Build project. Note that the active build configuration must be set to "Hardware".

Step 5: Putting the application on SD Card

There are two ways to write the image files on to the SD Card:

1. After the project is built successfully the Image and application files are packaged into the $platform_dir/dpu_trd_system/Hardware/package/sd_card.img directory. Now we can use applications such as Balena Etcher or dd utility to write the image into the SD Card. Mount the SD Card to the host machine and run the following.

$ cd $platform_dir/dpu_trd_system/Hardware/package/
$ sudo dd bs=4M if=sd_card.img of=$sd_card_dev status=progress conv=fsync

Note that $sd_card_dev is the mounted SD Card device (/dev/sd{x}). So this command needs to be changed to match the device name in your system. You can use "df -Th" command to see the mount point of your SD card.

2.An alternative way of writing the image on the SD Card is to create two partitions on the SD Card manually using this xilinx-wiki tutorial, copy the contents of the zcu104_custom_platform/dpu_trd_system/Hardware/package/sd_card/ directory to the boot partition and extract the $peta_prj/images/linux/rootfs.tar.gz file in to the second partition.

$ cd $platform_dir/dpu_trd_system/Hardware/package/sd_card/
$ cp ./* $sd_mount_point/boot/
$ cd $peta_prj/images/linux/
$ tar -xzvf rootfs.tar.gz -C $sd_mount_point/root/
$ sync

where $sd_mount_point is the SD card mount directory.

The second way is more suitable for debug and development phase where you often need to modify the root filesystem and/or application files.

Step 6: Running the application on target board

After putting the image on SD Card, attach it to the ZCU104 board, set the SW6 switch to SD Boot mode as below and turn the board power On. Refer to UG1267 for ZCU104 boot mode details.

Then connect to the board UART serial port and copy the SD card content to home directory.

root@petalinux:~# cp -r /media/sd-mmcblk0p1/data ~
root@petalinux:~# cp /media/sd-mmcblk0p1/rcan ~

Now we need to install the vitis-ai runtime.

root@petalinux:~# cd data/vitis-ai-runtime-1.4.0/2021.1/aarch64/centos/
root@petalinux:~# ./setup.sh
root@petalinux:~# cat /etc/vart.conf             #verify the installation 
firmware: /media/sd-mmcblk0p1/dpu.xclbin

Next we'll copy the model files to the /usr/share/vitis_ai_library/models/ folder.

root@petalinux:~# mkdir -p /usr/share/vitis_ai_library/models
root@petalinux:~# cp -r data/rcan_pruned_tf/ /usr/share/vitis_ai_library/models

Next we'll run the dpu_sw_optimize script. This script will:

• Auto-resize ext4 partition to the max capacity.
• Fine-tune QoS/Outstanding settings for DPU case to achieve a better performance.

root@petalinux:~# cd ~/data/
root@petalinux:~# tar -xzvf dpu_sw_optimize.tar.gz
root@petalinux:~# ./dpu_sw_optimize/zynqmp/zynqmp_dpu_optimize.sh
Auto resize ext4 partition ...[✔]
Start QoS config ...[✔]

Finally we'll run the application.

root@petalinux:~# cd ~ 
root@petalinux:~# cp data/img/butterfly.bmp ~
root@petalinux:~# ./rcan rcan_pruned_tf butterfly.bmp
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1111 12:06:21.787041  1158 demo.hpp:1183] batch: 0     image: butterfly.bmp

You should see that a "0_butterfly_result.jpg" file is created in the home directory. This is the result of the model. You can copy the result back to the host machine and open it.

$ scp root@BOARD_IP:~/0_butterfly_result.jpg $workspace

Known Issue #1: ZCU104 PMIC issue

As explained in the DPU TRD sectio 5.5.3, the IOUT_OC_FAULT_LIMIT on PMIC chip irps5401 is too low to afford DPU running. Therefore you may experience crash and reboot specially on running multiple threads of DPU. To solve this issue you need to apply the patch located in the dpu_sw_optimize.tar.gz file.

$ cd $workspace/Vitis-AI/dsa/DPU-TRD/app
$ tar -xzvf dpu_sw_optimize.tar.gz
$ cp -r dpu_sw_optimize/zynqmp/zcu104/recipes-kernel/ $peta_prj/project-spec/meta-user/
$ cd $peta_prj
$ petalinux-build

After the build is complete, you can copy the Linux component files to the corresponding Vitis Platform folders like as explained in step 1.

Conclusion

I hope this project helps you build your own AI applications on Xilinx-based Embedded platforms.

I would love to hear your thoughts and feedback on this project in the comment section. Also if you have any Ideas for future content you like to see, make sure to share it down in the comments. :)

Acknowledgments

I've been inspired by Mario Bergeron for creating some parts of this project. Make sure to checkout his cool projects on Xilinx embedded platforms.

#projectofthemonth