Accelerated App Development on Kria KD240 in Vitis 2023.2

Last month, I covered the basics in getting started with the new Kria KD240 Drives Starter Kit using the AMD FPGA tools with base project tutorials in Vivado, Vitis Unified, and PetaLinux versions 2023.2. And like I mentioned at the end of the PetaLinux tutorial, I would be using the resultant Linux image it created as a base for starting accelerated application design. So this project tutorial picks up there as I will be using that Linux image with the exported hardware platform from Vivado to create a simple vector addition accelerated application in the new Vitis Unified IDE.

Hardware acceleration in FPGA is the process of moving software tasks into the programmable logic of an FPGA. In an AMD FPGA like the Kria K24 SOM, this means selecting tasks running in software on one of the ARM-Cortex processor cores, converting the task's C/C++ code into RTL using HLS, and communicating with that RTL via a Linux-based OS kernel using an AXI interface. The simple vector addition example is the best to start with as there is an application component template for it within the Vitis Unified IDE to help users get familiar with the file structure and settings of accelerated development in Vitis.

Note: This tutorial is specific to the new Vitis Unified IDE (2023.2 and later) and is not backwards compatible with any previous Vitis Classic versions (2023.1 and earlier). Please refer back to my past accelerated application tutorials here, here, and here if using Vitis Classic.

Hardware Design in Vivado

As mentioned previously, the ARM processor in the Zynq MPSoC of the Kria SOM communicates with the offloaded task in the RTL via a Linux kernel using an AXI interface. This is why in the Vivado project, the option for the Vivado project to be an extensible Vitis platform has to be enabled so those AXI interfaces can be configured and included in the exported hardware platform (XSA) that is imported into Vitis to create the platform component with that the application component is then built on.

My project post for how to create a Vivado project for the Kria KD240 that is an extensible Vitis platform and configure those AXI ports is here. However, after some trial & error in the new Vitis Unified IDE I found a couple of settings that I needed to go back and tweak.

The first thing was an issue that I thought was relevant only to Vitis Classic, but it apparently has carried into Vitis Unified as well where the design checkpoint file (.dcp file) generated by Vivado during the synthesis process conflicts with the Vivado project Vitis builds/synthesizes for the offloaded software task when the accelerated application component is built. Therefore, there must be no design checkpoint file from Vivado within the exported XSA.

To stop Vivado from creating any design checkpoint files there are two settings to change in Vivado.

The first of which is to disable incremental synthesis. Select Settings from the Flow Navigator window and under Synthesis > Incremental synthesis, change it to Disable incremental synthesis:

The second setting is to generate the block design with the Global synthesis option selected (select Generate Block Design from the Flow Navigator window):

The next thing I found was that Vitis needed an extra bitstream file included in the XSA that didn't have the extra extra header configuration that a.bit file contains, so a.bin of the bitstream was needed. To tell Vivado to generate a.bin version of the bitstream along with the.bit version, enable -bin_file under Bitstream in Settings of Vivado:

Finally, I discovered a new quirk in Vitis Unified as compared to Vitis Classic. In Vitis Classic, you could have extra AXI ports enabled in the Platform in Vivado that weren't connected in the final accelerated application as a way of future-proofing so it wasn't necessary to rebuild the Vivado project and re-export a new XSA in the event they were needed later on. However, having unconnected AXI ports in the Platform in Vitis Unified causes a build error in the accelerated application component.

In my original post, I had enabled 8 regular AXI interfaces through the AXI interconnect as options for future applications. So I had to go back and disable them for the vector addition accelerated application since it didn't use them:

Finally I re-build the Vivado project (ran synthesis, implementation, and generate bitstream) and re-exported the platform with the same name/settings as before (see those steps here) to get the XSA needed for the vector addition accelerated application.

Add Filesystem Dependencies in PetaLinux

I covered the basics of getting started with the KD240 in PetaLinux 2023.2 already, so I'm starting with that PetaLinux project built from the KD240 BSP in my previous project post here.

Note: It is important to run a build on that PetaLinux project before adding the following packages or the build will take quite a while.

Since I had to generate/export a new XSA file, I also pulling it into the PetaLinux project as well:

~/Kria_KD240/kd240_linux_os$ petalinux-config --get-hw-description ./<path to new XSA>

While the PetaLinux project was created using the BSP for the Kria KD240, there are still some extra filesystem libraries and package groups that need to be added for acceleration specifically.

Launch the root filesystem configuration editor:

~/Kria_KD240/kd240_linux_os$ petalinux-config -c rootfs

Then enable the following:

Filesystem Packages --> console --> utils --> git --> [*] git
Filesystem Packages --> base --> dnf --> [*] dnf
Filesystem Packages --> x11 --> base --> libdrm --> [*] libdrm
Filesystem Packages --> x11 --> base --> libdrm --> [*] libdrm-tests
Filesystem Packages --> libs --> xrt --> [*] xrt
Filesystem Packages --> libs --> xrt --> [*] xrt-dev
Filesystem Packages --> libs --> zocl --> [*] zocl 
Filesystem Packages --> libs --> opencl-headers --> [*] opencl-headers 
Filesystem Packages --> libs --> opencl-clhpp --> [*] opencl-clhpp-dev
Petaliunx Package Groups --> packagegroup-petalinux --> [*] packagegroup-petalinux
Petaliunx Package Groups --> packagegroup-petalinux-gstreamer --> [*] packagegroup-petalinux-gstreamer
Petaliunx Package Groups --> packagegroup-petalinux-opencv --> [*] packagegroup-petalinux-opencv
Petaliunx Package Groups --> packagegroup-petalinux-v4lutils --> [*] packagegroup-petalinux-v4lutils
Petaliunx Package Groups --> packagegroup-petalinux-vitis-acceleration-essential --> [*] packagegroup-petalinux-vitis-acceleration-essential
Petaliunx Package Groups --> packagegroup-petalinux-x11 --> [*] packagegroup-petalinux-x11

PetaLinux 2023.2 introduces a new package group I noticed specifically for acceleration development in vitis called packagegroup-petalinux-vitis-acceleration-essential. Which does enable most of what I had previously been enabling manually, minus few things:

After importing the updated XSA and enabling the necessary filesystem libraries and package groups, rebuild the project:

~/Kria_KD240/kd240_linux_os$ petalinux-build

And create a new wic image, as well as boot binary:

~/Kria_KD240/kd240_linux_os$ petalinux-package --boot --u-boot --force
~/Kria_KD240/kd240_linux_os$ petalinux-package --wic --images-dir images/linux/ --bootfiles "ramdisk.cpio.gz.u-boot,boot.scr,Image,system.dtb,system-zynqmp-sck-kd-g-revA.dtb"

Finally, flash an SD card with the wic image:

And flash the new boot binary onto the Kria K24 SOM. I personally like to use the Boot Image Recovery Tool to guarantee I won't mess it up (instructions here).

Build SDK for Sysroot

In order to build the accelerated application for the Linux OS running on the Kria, Vitis Unified needs a local copy of that Linux OS in the form of a sysroot. This is included in the SDK of a PetaLinux project.

In order to create an SDK of a PetaLinux project, run another build with the --sdk flag:

~/Kria_KD240/kd240_linux_os$ petalinux-build --sdk

This will output a script file in the /<PetaLinux project directory>/images/linux directory that can be used to install the SDK (and therefore sysroot) into a location where Vitis can then use it.

Custom Platform Directory Structure

As I mentioned, Vitis Unified needs a local copy of the Linux OS that will be running on the Kria K24 SOM as well as the corresponding boot files to be able to build the accelerated application. I've tried just pointing it all to the /<PetaLinux project directory>/images/linux directory, but that ends up causing errors since different versions of the same boot files and root filesystem exist there.

Instead, I create a directory structure to install the sysroot and relevant boot files into that I point Vitis to (done in later steps).

Within the top level directory of the corresponding Vivado project directory (the Kria_KD240 directory), I create a "custom_platform" directory for the sysroot, with a "pfm" subdirectory that contains a "boot" and "sd_dir" for the relevant boot files:

~$ cd ./Kria_KD240/
~/Kria_KD240$ mkdir -p kd240_custom_platform
~/Kria_KD240$ cd ./kd240_custom_platform/
~/Kria_KD240/kd240_custom_platform$ mkdir -p pfm
~/Kria_KD240/kd240_custom_platform$ cd ./pfm/
~/Kria_KD240/kd240_custom_platform/pfm$ mkdir -p boot
~/Kria_KD240/kd240_custom_platform/pfm$ mkdir -p sd_dir

I then copy the relevant boot files into the "boot" directory from the aforementioned /<PetaLinux project directory>/images/linux directory:

~/Kria_KD240/kd240_custom_platform/pfm$ cd ../../kd240_linux_os/images/linux/
~/Kria_KD240/kd240_linux_os/images/linux$ cp ./boot.scr ../../../kd240_custom_platform/pfm/boot/
~/Kria_KD240/kd240_linux_os/images/linux$ cp ./bl31.elf ../../../kd240_custom_platform/pfm/boot/
~/Kria_KD240/kd240_linux_os/images/linux$ cp ./pmufw.elf ../../../kd240_custom_platform/pfm/boot/
~/Kria_KD240/kd240_linux_os/images/linux$ cp ./system.dtb ../../../kd240_custom_platform/pfm/boot/
~/Kria_KD240/kd240_linux_os/images/linux$ cp ./u-boot.elf ../../../kd240_custom_platform/pfm/boot/
~/Kria_KD240/kd240_linux_os/images/linux$ cp ./zynqmp_fsbl.elf ../../../kd240_custom_platform/pfm/boot/

And install the SDK (sysroot) into the "custom_platform" directory:

~/Kria_KD240/kd240_linux_os/images/linux$ source /tools/Xilinx/PetaLinux/2023.2/settings.sh
~/Kria_KD240/kd240_linux_os/images/linux$ ./sdk.sh -d ../../../kd240_custom_platform/

Keep in mind, this directory structure doesn't necessarily need to be copied exactly. The important thing is that a directory containing only one copy and one version of the relevant boot files exists with noting else in it. And they are not in the "boot" directory of the sysroot.

Generate Device Tree Overlay with XSCT

Since accelerated applications are implemented by means of partial reconfiguration in AMD FPGAs, that means the bitstream containing the offloaded task is not flashed onto the programmable logic (PL) until after the Linux OS is booted and running. Therefore, the corresponding device tree for it has to be implemented as a device tree overlay where the device tree blob for the PL is deployed when the accelerated application is loaded.

The easiest way I've found to create the device tree device tree blob for the PL is to first create the device tree source file (.dtsi) using the Device Tree Generator (DTG) from the AMD Software Command-Line Tools (XSCT):

~/Kria_KD240/kd240_custom_platform$ source /tools/Xilinx/Vitis/2023.2/settings64.sh
~/Kria_KD240/kd240_custom_platform$ xsct
xsct% hsi::open_hw_design ../kd240_base.xsa 
xsct% createdts -hw ../kd240_base.xsa -zocl -platform-name kria_kd240 -git-branch xlnx_rel_v2023.2 -overlay -compile -out ./dtg_output
xsct% exit

I saw the nice big warning text that XSCT is will be going away at some point in future versions of Vitis, but I'm going to keep using it until it's gone. I also never figured out what any of the connection warnings were, but they didn't seem to cause any issues later on so I ignored them.

Then just compile the device tree source file (.dtsi) into the device tree blob (.dtbo) using the plain old device tree compiler (dtc):

~/Kria_KD240/kd240_custom_platform$ cd ./dtg_output/dtg_output/kria_kd240/psu_cortexa53_0/device_tree_domain/bsp/
~/Kria_KD240/kd240_custom_platform/dtg_output/dtg_output/kria_kd240/psu_cortexa53_0/device_tree_domain/bsp$ dtc -@ -O dtb -o pl.dtbo pl.dtsi

This device tree blob is what will be transferred to the Kria KD240 to be used with the accelerated application is loaded. But is it NOT the device tree file used in the aforementioned "boot" directory that Vitis is using as a local copy of the Kria's Linux image for compiling/building the accelerated application.

Create Accelerated Platform Component in Vitis IDE

Finally, everything is in place to create the accelerated application in Vitis. I'm using the same Vitis workspace from my previous tutorial where I created a bare-metal (no-OS) C application for the KD240 since it's possible to have multiple platform components and application components in the same workspace.

The first step in Vitis is to create a new platform component based on the XSA exported from Vivado that is an extensible Vitis platform.

Since I already had a platform component in this workspace built on an XSA that was exported from a Vivado project that is NOT an extensible Vitis platform, I specified "accel" in the name in this platform:

Since I selected the option for both hardware and hardware emulation when I exported the platform in Vivado, I specified the XSA for both in the platform component as well:

Select Linus as the operating system and enable the option for Vitis to generate the boot components:

Review the selections to verify and click Finish to generate the platform component.

After the platform component is generated, it needs to be pointed to the sysroot and relevant boot files placed in that "custom_platform" directory in the previous steps.

Open vitis-comp.json under platform component name > psu_cortexa53 > Linux on psu_cortexa53 and make the following changes (I have my custom file structure here, but replace with your own if different):

• Display Name: xrt
• Bif File: select option to generate one

The little icon next to the Browse button for Bif file is the button to generate a new Bif file:

• Pre-Built Image Directory: ~/Kria_KD240/kd240_custom_platform/pfm/boot
• DTB File: ~/Kria_KD240/kd240_custom_platform/pfm/boot/system.dtb
• FAT32 Partition Directory: ~/Kria_KD240/kd240_custom_platform/pfm/sd_dir
• Qemu Data: ~/Kria_KD240/kd240_custom_platform/pfm/boot
• Qemu Args File: Leave default value that was auto-populated
• Pmu Args File: Leave default value that was auto-populated

Then run a build of the platform component (select it as the target component in the Flow window and click Build):

Create Accelerated Application Component in Vitis IDE

The application component template for the vector addition accelerated application (vadd) is located in a repository in the Vitis Examples that may or may not be downloaded already depending on the installation settings chosen when installing Vitis.

From the Welcome window, select Examples. Then under Acceleration Examples, if the download button appears next to Vitis Accelerated Libraries Repository, click it and wait until it completes.

Then select Simple Vector Addition under Installed Examples Repository, and Create Application from Template:

I left the default name as vadd, but you can change it if desired:

Select the platform created in the previous steps:

Specify the Image from the /<PetaLinux project directory>/images/linux directory as the kernel image (this is different than the sysroot and relevant boot files in the custom_platform directory). For the root filesystem, specify rootfs.ext4 from the /<PetaLinux project directory>/images/linux directory. Then for the sysroot point it to the environment-setup-cortexa72-cortexa53-xilinx-linux directory within custom_platform directory (~/Kria_KD240/kd240_custom_platform/environment-setup-cortexa72-cortexa53-xilinx-linux in my case):

Review the selections to verify and click Finish to generate the vadd application component.

Wait for all of the components to generate, which is the accelerated kernel, host application, and top level vadd component:

In the vadd component (the component with the sole name "vadd") under Sources > package, open package.cfg.

Since the Kria SOM is bootstrapped to QSPI boot mode, the Boot mode under General needs to be changed from SD to QSPI:

Then under Linux boot, it needs to be pointed to all of the same relevant boot files in the "custom_platform" directory that the platform component was and the kernel image and root filesystem file in /<PetaLinux project directory>/images/linux. So in my case that ended up being:

• ARM trusted firmware elf: ~/Kria_KD240/kd240_custom_platform/pfm/boot/pmufw.elf
• DTB file: ~/Kria_KD240/kd240_custom_platform/pfm/boot/system.dtb
• U-boot ELF: ~/Kria_KD240/kd240_custom_platform/pfm/boot/u-boot.elf
• Image format: ext4
• Kernel Image: /<PetaLinux project directory>/images/linux/Image
• Root file system: /<PetaLinux project directory>/images/linux/rootfs.ext4

Select vadd as the target component in the Flow window, then under Hardware, click Build All, a prompt will appear asking to also build the the accelerated kernel and host application components:

The build will take a few minutes:

And as long as the terminal in Vitis says Build Finished Successfully, the red x next to Build All in the Flow window can be ignored. But also verify there is a green check mark next to Build Binary Container under LINK binary_container_1:

I scrolled back through the Vitis terminal to see what the errors were, it appears as though it was trying to run something on the Kria hardware itself. And even though I had my KD240 powered on and connected to my PC, because it's in bootstrapped to QSPI boot mode by hardware (thus has to be put in JTAG boot mode manually) Vitis is unable to launch a run on it directly. Which is why it shows errors but the build still completes successfully.

Disclaimer: this is just a guess, I have not confirmed this officially. I will update this if I find different.

I also wanted to make a note that if you want to make changes to the C++ code of the host application running on the ARM-core, the source file is located under vadd_host > Sources > vadd.cpp

And the source file for the accelerated kernel that's running the the programmable logic (the task being converted to RTL with HLS) is located under vadd_krnl_vadd > Sources > krnl_vadd.cpp if you want to make any changes there. I'm not making any changes in this post just because it's already turned into a novel...

Transfer Files to KD240

With the vadd accelerated application completed, the relevant output files can be transferred to the KD240.

I personally like to create a folder with my "custom platform" directory of the files to transfer to the target Kria board just so everything is in one easy place I can call from the secure transfer command (scp):

~/Kria_KD240/kd240_custom_platform$ mkdir -p vadd
~/Kria_KD240/kd240_custom_platform$ cd ./vadd/

This is where I also create the description file, shell.json, for the accelerated application:

~/Kria_KD240/kd240_custom_platform/vadd$ gedit shell.json

And copy+paste the following into it (pulled from another example):

{
  "shell_type" : "XRT_FLAT",
  "num_slots": "1"
}

I then copy the device tree blob, host application, and binary container for the accelerated kernel into this directory:

~/Kria_KD240/kd240_custom_platform/vadd$ cp ../dtg_output/dtg_output/kria_kd240/psu_cortexa53_0/device_tree_domain/bsp/pl.dtbo ./
~/Kria_KD240/kd240_custom_platform/vadd$ cp ../../vitis_workspace/vadd_host/build/hw/vadd_host ./
~/Kria_KD240/kd240_custom_platform/vadd$ cp ../../vitis_workspace/vadd/build/hw/hw_link/binary_container_1.xclbin ./

The binary container for the accelerated kernel needs to have its file extension changed from.xclbin to just.bin:

~/Kria_KD240/kd240_custom_platform/vadd$ mv binary_container_1.xclbin binary_container_1.bin

Since my KD240 is connected to my router via Ethernet, I use the the secure transfer command (scp) to transfer the final accelerated application files to it:

~/Kria_KD240/kd240_custom_platform/vadd$ ping 192.168.1.xxx
~/Kria_KD240/kd240_custom_platform/vadd$ scp binary_container_1.bin pl.dtbo shell.json vadd_host petalinux@192.168.1.xxx:/home/petalinux

Run Accelerated Application on KD240

To run the accelerated application on the KD240, a new directory for it first needs to be created in /lib/firmware/xilinx then the kernel's binary container, device tree blob, and json description file copied to it:

xilinx-kd240-starterkit-20232:~$ sudo mkdir /lib/firmware/xilinx/vadd
xilinx-kd240-starterkit-20232:~$ sudo cp binary_container_1.bin pl.dtbo shell.json /lib/firmware/xilinx/vadd

Then use the xmutil command to first list the available accelerated applications (vadd will now appear), unload the default application, then load the vadd accelerated application:

xilinx-kd240-starterkit-20232:~$ sudo xmutil listapps
xilinx-kd240-starterkit-20232:~$ sudo xmutil unloadapp
xilinx-kd240-starterkit-20232:~$ sudo xmutil loadapp vadd

Make the vadd host application file (vadd_host) executable and run it with the appropriate kernel binary container (this is why the files were just copied to the /lib/firmware/xilinx/vadd directory instead of moved there):

xilinx-kd240-starterkit-20232:~$ chmod +x vadd_host
xilinx-kd240-starterkit-20232:~$ ./vadd_host binary_container_1.bin

Some warnings will appear that memory leaks will occur if anything attempts to use the new device tree nodes introduced by the overlay in pl.dtbo after the vadd application is unloaded as a reminder.

But overall, you should see TEST PASSES if the vadd accelerated application was able execute properly and return the sum of two vector values from the programmable logic of the Zynq MPSoC of the Kria:

With the basics covered, I plan to follow this up with a customized accelerated application since there are some more settings changes involved there, but this post is long enough with the bare necessities!