Updating the HW Design in an Accelerated App in Vitis 2023.2

Over the past few years that I've been developing hardware accelerated applications for AMD FPGAs using Vitis, I've come to find that an easy way to really mess up a Vitis workspace is when I am updating the referenced XSA file because I had to make some change in the hardware design in Vivado. Since the XSA is referenced both by Vitis in the platform component/project and the PetaLinux project that creates the sysroot the application component/project in Vitis uses, it is all too easy to miss a reference and end up with a wonky workspace that is secretly trying to build on two different versions of your exported XSA file.

Now that I've been spending time adjusting to the new VS Code-based version of Vitis with the release of 2023.2 in my previous post of how to create/build the accelerated vector addition example for my Kria KD240, I decided that it was worth writing up my tried and true method for updating the referenced XSA in a Vitis workspace for a Linux-based application. This is also because I did have to modify the hardware design in Vivado a couple of times as I found differences in the new Vitis Unified workflow compared to how the Eclipse-based Vitis Classic worked.

Re-Export XSA from Vivado

As I've detailed in my previous post for how to export the XSA hardware platform from Vivado, be sure to check the option to include the bitstream and whether you want support for just hardware or both hardware and hardware emulation.

I usually keep the same name and directory location for the exported XSA file, but don't let that fool you: neither Vitis Unified nor Vitis Classic will re-read it to detect the new changes in it without performing the following steps.

Update PetaLinux HW Design

Once the new XSA file has been exported from Vivado, the PetaLinux project that is building the Linux image for the target FPGA that the accelerated application is ultimately running in needs to be updated next before hoping back over into Vitis.

Change directories into the target PetaLinux project the accelerated application will run in and import the new XSA hardware file:

~$ petalinux-config --get-hw-description ./<path to new XSA>

This will reopen the hardware configuration ASCII GUI. Exit and chose to save the new configuration (which is the new XSA), then rebuild the project:

~$ petalinux-build

Once the project has been built, build a new sysroot (SDK) for Vitis to use for the accelerated application:

~$ petalinux-build --sdk

Replace Sysroot in Custom Platform Directory

In my last post about how to create/build the accelerated vector addition example for my Kria KD240, I outlined how I create a custom platform directory for the sysroot and relevant boot files that Vitis references when building the accelerated application (since the accelerated application is Linux-based). So the secondary step after updating the PetaLinux project with the new XSA and building the new sysroot is to export it to this custom platform directory that Vitis is pointed to in the configuration settings of both the platform component and application component.

You could totally export the new sysroot to a new location and point the platform component and application component in Vitis to it. However, since there are multiple locations in the Vitis settings that need to point to this same sysroot directory, I find it easier (and less error prone) to delete the old sysroot files from the same directory:

~$ cd ./FPGA_custom_platform 
~/FPGA_custom_platform$ rm -rf ./sysroots
~/FPGA_custom_platform$ rm ./environment-setup-cortexa72-cortexa53-xilinx-linux
~/FPGA_custom_platform$ rm ./site-config-cortexa72-cortexa53-xilinx-linux
~/FPGA_custom_platform$ rm ./version-cortexa72-cortexa53-xilinx-linux

And extract the new sysroot to it:

~/<PetaLinux prj dir>/images/linux$ ./sdk.sh -d /<path to>/FPGA_custom_platform/

Replace Boot Files in Custom Platform Directory

Along with the updated sysroot, the old versions of the relevant boot files copied from the PetaLinux project also need to be deleted:

~/FPGA_custom_platform$ rm ./pfm/boot/boot.scr
~/FPGA_custom_platform$ rm ./pfm/boot/bl31.elf
~/FPGA_custom_platform$ rm ./pfm/boot/pmufw.elf
~/FPGA_custom_platform$ rm ./pfm/boot/system.dtb
~/FPGA_custom_platform$ rm ./pfm/boot/u-boot.elf
~/FPGA_custom_platform$ rm ./pfm/boot/zynqmp_fsbl.elf

Then their updated versions copied back over from the PetaLinux project output directory:

~/FPGA_custom_platform$ cd /<PetaLinux prj dir>/images/linux/
~/<PetaLinux prj dir>/images/linux$ cp ./boot.scr ~/FPGA_custom_platform/pfm/boot/
~/<PetaLinux prj dir>s/images/linux$ cp ./bl31.elf ~/FPGA_custom_platform/pfm/boot/
~/<PetaLinux prj dir>/images/linux$ cp ./pmufw.elf ~/FPGA_custom_platform/pfm/boot/
~/<PetaLinux prj dir>/images/linux$ cp ./system.dtb ~/FPGA_custom_platform/pfm/boot/
~/<PetaLinux prj dir>/images/linux$ cp ./u-boot.elf ~/FPGA_custom_platform/pfm/boot/
~/<PetaLinux prj dir>/images/linux$ cp ./zynqmp_fsbl.elf ~/FPGA_custom_platform/pfm/boot/

Again, it's worth noting you cannot just point Vitis to the PetaLinux project output directory of ./images/linux because multiple versions of the boot files and root filesystem exist there and causes issues when running a build (and yes, I have tried and learned this the hard way).

Generate New Device Tree Overlay (If Relevant)

As I mentioned in my previous project post, due to the fact that the bitstream containing the offloaded task (aka the "accelerated" task) is not flashed onto the programmable logic (PL) until after the Linux OS is booted and running, a device tree overlay with the relevant device tree nodes must be created for it.

Unless the changes in the hardware design in Vivado create changes/additions/deletions to the corresponding device tree node(s), there is no need to create a new device tree overlay. However, the steps are pretty straight forward to recreate it.

I personally like to delete the previous version since I create it within my custom platform directory:

~/FPGA_custom_platform$ rm -rf ./dtg_output

Then follow the exact same steps in XSCT to create a new device tree overlay:

~/FPGA_custom_platform$ source /tools/Xilinx/Vitis/2023.2/settings64.sh
~/FPGA_custom_platform$ xsct
xsct% hsi::open_hw_design /<path to>/fpga_base.xsa 
xsct% createdts -hw /<path to>/fpga_base.xsa -zocl -platform-name fpga_platform -git-branch xlnx_rel_v2023.2 -overlay -compile -out ./dtg_output
xsct% exit
~/FPGA_custom_platform$ cd ./dtg_output/dtg_output/fpga_platform/psu_cortexa53_0/device_tree_domain/bsp/
~/FPGA_custom_platform/dtg_output/dtg_output/fpga_platform/psu_cortexa53_0/device_tree_domain/bsp$ dtc -@ -O dtb -o pl.dtbo pl.dtsi

Again, you can create a new device tree overlay in a new location if desired. Since Vitis isn't pointing to this device tree overlay file and it's just used on the target FPGA when loading the accelerated application, it's not quite the same as the sysroot.

Update Platform Component in Vitis

With all of the referenced elements updated with the new XSA file, it's finally time to update Vitis to pull in the new XSA file.

Starting with the platform component since any application component is built based on the platform component, open the vitis-comp.json file in the Settings folder for the platform component.

Then under Hardware Specification for the platform, select Switch XSA and browse to the new XSA file.

Under SoftwareSpecification for the platform, select Update Emulation XSA and browse to the new XSA file (if the option for hardware emulation was selected when it was exported from Vivado).

The platform component will then show as out-of-date. Under the Flow window for the platform component, hover over the "build" option and click the broom icon to clean the platform component, then click Build All to rebuild it.

Rebuild Accelerated Application Component

Because I simply updated the files in the same custom platform directory for the sysroot and relevant boot files instead of a new directory location, the last step is to clean and rebuild the accelerated application.

Again hover over the "build" option for the component in the Flow window and click the broom icon to clean the component, then click Build All to rebuild the accelerated application.

At this point, the workflow is the same to transfer the newly updated files to the target FPGA as I outlined in my aforementioned project post. Hopefully this small detour for updating the hardware design in an accelerated application in a Vitis workspace is helpful!