Accelerated Design Development on Kria KR260 in Vitis 2022.1

Since the workflow for developing an accelerated application has updated a bit in Vitis 2022.1 as compared to Vitis 2021.x and the Kria KR260 has also found its way into many of my current/future projects, I thought it would be worth a quick write-up demonstrating how to compile the Vector Addition accelerated application for the KR260 and run it on KR260 hardware. I will then expand on this and use the Vector Addition project as a template project for my planned custom accelerated application for the KR260.

This project write-up assumes all previous knowledge conveyed in my previous KR260 project posts. Also, this workflow is only applicable to 2022.1, please see my accelerated application flow for the Kria KV260 for the 2021.x workflow.

Hardware Design in Vivado

The first step in creating any custom Vitis Platform and acceleration application is to create the base hardware design in Vivado. For the KR260, I’ve already covered how to create a fairly all-encompassing hardware design in Vivado 2022.1 in two previous projects posts here and here. I am starting with the hardware design in this project post that resulted from my second post with all of the KR260’s peripherals enabled in the ZynqMP’s IP in the block design.

The overall result of the Vivado project is an exported hardware platform to pull into PetaLinux and Vitis to build the software on (kria_base.xsa in the Vivado project top level directory for me).

Linux Image in PetaLinux

Once the hardware design is complete, the embedded Linux image the accelerated application will be running on for the target board (the KR260 in this case) with the proper libraries and dependencies needs to be created. Again, I’ve already covered all of the detail of how to create this PetaLinux project for the KR260 in a previous post here.

While the XRT and ZOCL packages should be enabled by default in a PetaLinux project created using the KR260 BSP, it’s worth launching the root filesystem configuration editor to double-check before running a build:

~/Kria_KR260/linux_os$ petalinux-config -c rootfs 
---> Filesystem Packages
------> libs
---------> xrt
------------ [*] xrt
------------ [*] xrt-dev
---------> zocl
------------ [*] zocl

Custom Vitis Platform Directory Structure

With the hardware design complete in Vivado and the embedded Linux image successfully built in PetaLinux, the next step is to create the directory structure for the custom Vitis platform.

You can create this in pretty much any preferred location, but I still like my tried and true location of the corresponding Vivado project top level directory. If you recall from my previous post, that’s the Kria_KR260 directory for me. After changing directories into it, create a directory for the custom platform with pfm sub-directory. Then within the pfm directory, create the boot and sd_dir directories:

~$ cd ./Kria_KR260/
~/Kria_KR260$ mkdir -p kr260_custom_platform
~/Kria_KR260$ cd ./kr260_custom_platform/
~/Kria_KR260/kr260_custom_platform$ mkdir -p pfm
~/Kria_KR260/kr260_custom_platform$ cd ./pfm/
~/Kria_KR260/kr260_custom_platform/pfm$ mkdir -p boot
~/Kria_KR260/kr260_custom_platform/pfm$ mkdir -p sd_dir

Change directories back into the PetaLinux output directory, <PetaLinux project>/images/linux, and copy bl31.elf, pmufw.elf, system.dtb, u-boot.elf, and zynqmp_fsbl.elf into the boot directory of the custom platform.

~/Kria_KR260/kr260_custom_platform/pfm$ cd ../../linux_os/images/linux/
~/Kria_KR260/linux_os/images/linux$ cp ./bl31.elf ../../../kr260_custom_platform/pfm/boot/
~/Kria_KR260/linux_os/images/linux$ cp ./pmufw.elf ../../../kr260_custom_platform/pfm/boot/
~/Kria_KR260/linux_os/images/linux$ cp ./system.dtb ../../../kr260_custom_platform/pfm/boot/
~/Kria_KR260/linux_os/images/linux$ cp ./u-boot.elf ../../../kr260_custom_platform/pfm/boot/
~/Kria_KR260/linux_os/images/linux$ cp ./zynqmp_fsbl.elf ../../../kr260_custom_platform/pfm/boot/

Next, install the generated SDK from the PetaLinux project in the custom platform’s top level directory. This is so Vitis has the sysroot available with the appropriate libraries and C files to build the accelerated application.

~/Kria_KR260/linux_os/images/linux$ source /tools/Xilinx/PetaLinux/2022.1/settings.sh
~/Kria_KR260/linux_os/images/linux$ ./sdk.sh -d ../../../kr260_custom_platform/

Once the SDK has successfully been installed, change directories back into the custom platform:

~/Kria_KR260/linux_os/images/linux$ cd ../../../kr260_custom_platform/

Generate Device Tree Overlay with XSCT

One thing I’m pretty happy with in Vitis 2022.1 is that they’ve streamlined the process for generating device tree source files in the Xilinx Software Command Line Tool (XSCT) by adding the createdts command. Source the Vitis toolset in your environment then launch XSCT:

~/Kria_KR260/kr260_custom_platform$ source /tools/Xilinx/Vitis/2022.1/settings64.sh
~/Kria_KR260/kr260_custom_platform$ xsct

Now in theory, you shouldn’t need to manually open the hardware design before running the createdts command, but alas that wasn’t the case for me. Also, if you run into any errors, exit XSCT and relaunch it otherwise it won’t actually attempt to recompile.

In the createdts command, the option to enable ZOCL (-zocl) and specify that it’s an overlay (-overlay) are the main objectives. Give it any desired platform name (-platform-name <name>) and output directory (-out <directory>).

xsct% hsi::open_hw_design ../kria_base.xsa 
xsct% createdts -hw ../kria_base.xsa -zocl -platform-name kria_kr260 -git-branch xlnx_rel_v2022.1 -overlay -compile -out ./dtg_output
xsct% exit

Once the device tree source files have been generated, compile them using the standard Linux device tree compiler, dtg.

Launch Vitis and Create Platform Project

With all of the hardware and software components needed, launch the Vitis IDE and select the custom platform top level directory as the workspace (which is ~/Kria_KR260/kr260_custom_platform for me):

~$ source /tools/Xilinx/Vitis/2022.1/settings64.sh
~$ vitis

When the new blank workspace is generated, select the option to create a new Platform Project.

Give it any desired name, I chose kr260_custom.

Then select the exported XSA file from the Vivado project that’s been used for everything thus far such as the PetaLinux project and device tree compiler.

Set the OS to Linux and the processor to psu_cortexa53 (which should be the default processor set) with 64-bit architecture. Uncheck the option to Generate boot components as we’ll be pointing to the boot directory created manually in previous steps. Then click Finish to generate the platform project.

Once the Platform project generates, the platform.spr file will open automatically. Navigate to kr260_custom > psu_cortexa53 > linux on psu_cortexa53.

Use the drop down menu next to the Bif file line and select the option to Generate Bif.

Then set Boot Components Directory to the boot directory created in earlier steps: ~/Kria_KR260/kr260_custom_platform/pfm/boot

Set FAT32 Partition Directory to the sd_dir directory created in earlier steps: ~/Kria_KR260/kr260_custom_platform/pfm/sd_dir

Build the Platform Project by right-clicking on its name in the Explorer window and selecting the option to Build Project.

Create Application Project

With the Platform Project complete, select New > Application Project to create the application project. Select the Platform Project as the target platform for the application.

Give the application the desired name, I named it kr260_vadd:

Set the Sysroot path to the sysroot created when the SDK was installed in the custom platform directory: ./kr260_custom_platform/sysroots/cortexa72-cortexa53-xilinx-linux.

Then select rootfs.ext4 from the output directory of the PetaLinux project (./<PetaLinux project/images/linux) for the Root FS, and the Image file from the same location for Kernel Image.

Finally, select the Vector Addition application project template under Acceleration templates with PL and AIE accelerators.

Side note: Vitis 2022.1 doesn’t like dark mode in Ubuntu 22.04. I will remember to change it to light mode next time before taking screenshots for my write-ups.

Upload Accelerated Application Files to KR260

Boot the Kria KR260 board with the embedded Linux image generated by the PetaLinux project and get it connected to the same local network as the host PC as outlined in my previous post here.

I found it’s easiest to create a directory to copy all of the files I need to transfer to the KR260 into. The files needed are the device tree blob (pl.dtbo), acceleration binary container for XRT configuration that contains the FPGA bitstream and metadata that describes the kernels is located in (binary_container_1.xclbin), the description file (shell.json), and compiled application file (kr260_vadd):

~/Kria_KR260/kr260_custom_platform$ mkdir -p vadd_file_transfer
~/Kria_KR260/kr260_custom_platform$ cd ./vadd_file_transfer/
~/Kria_KR260/kr260_custom_platform/vadd_file_transfer$ cp ../dtg_output/dtg_output/kria_kr260/psu_cortexa53_0/device_tree_domain/bsp/pl.dtbo ./
~/Kria_KR260/kr260_custom_platform/vadd_file_transfer$ cp ../vadd_system/Hardware/binary_container_1.xclbin ./
~/Kria_KR260/kr260_custom_platform/vadd_file_transfer$ cp ../kr260_vadd/Hardware/kr260_vadd ./

Next, create the description file, shell.json, for the kr260_vadd application:

~/Kria_KR260/kr260_custom_platform/vadd_file_transfer$ gedit shell.json

And copy+paste the following into it:

{
  "shell_type" : "XRT_FLAT",
  "num_slots": "1"
}

Then use a file transfer command such as scp to upload the vadd accelerated application files to the KR260:

~/Kria_KR260/kr260_custom_platform/vadd_file_transfer$ scp pl.dtbo binary_container_1.xclbin shell.json kr260_vadd petalinux@<KR260 IP>:/home/petalinux

Once successfully uploaded, copy them from the home directory to the proper firmware directory in the root (since you can file transfer straight to any root file):

xilinx-kr260-starterkit-20221:~$ sudo mkdir /lib/firmware/xilinx/kr260_vadd

Before moving the device tree overlay, description file, and binary container file to the accelerated application’s folder in the Linux firmware directory, it’s important to change the extension of the binary container file from .xclbin to .bin. Otherwise XRT won’t be able to extract the FPGA bitstream out of it to flash the programmable logic of the Kria with.

xilinx-kr260-starterkit-20221:~$ mv binary_container_1.xclbin binary_container_1.bin
xilinx-kr260-starterkit-20221:~$ cp pl.dtbo binary_container_1.bin shell.json /lib/firmware/xilinx/kr260_vadd

After copying the vector addition application’s files to its Linux firmware directory, unload the current accelerated application on the KR260 and load the vector addition application:

xilinx-kr260-starterkit-20221:~$ sudo xmutil listapps
xilinx-kr260-starterkit-20221:~$ sudo xmutil unloadapp
xilinx-kr260-starterkit-20221:~$ sudo xmutil loadapp kr260_vadd

Not changing the file extension of the binary container file from .xclbin to .bin results in an accelerated application that will not load to slot 0 (it will stay inactive by loading to slot -1) and if you try to run it you’ll get an error like this:

[XRT] ERROR: No devices found
Error: Unable to find Target Device

Test Accelerated Application

Run the vector addition application to test it. First make it executable then run it with the binary container file passed as an argument.

xilinx-kr260-starterkit-20221:~$ chmod +x ./kr260_vadd
xilinx-kr260-starterkit-20221:~$ ./kr260_vadd binary_container_1.bin

It should give a print out indicating that the vector addition application has ran successfully:

xilinx-kr260-starterkit-20221:~$ ./kr260_vadd binary_container_1.bin 
INFO: Reading binary_container_1.bin
Loading: 'binary_container_1.bin'
Trying to program device[0]: edge
Device[0]: program successful!
TEST PASSED

And that’s it!