Hardware components | ||||||
 |
| × | 1 | |||
 |
| × | 1 | |||
Software apps and online services | ||||||
 |
| |||||
 |
| |||||
This project is part 2 of a 4 part series of projects, where we will progressively create a Vitis-AI and ROS2 enabled platform for Ultra96-V2:
- Part 1 : Building the foundational designs
- Part 2 : Combining designs into a common platform
- Part 3 : Adding support for Vitis-AI
- Part 4 : Adding support for ROS2
The motivation of this series of projects is to enable users to create their own custom AI applications.
Introduction - Part IIIn the previous project ( http://avnet.me/u96v2-sbc-dualcam-2022.2 ),we learned how to build the foundational designs for Ultra96-V2:
- u96v2_sbc_base
- u96v2_sbc_dualcam
We built an SD image for each design and learned to use each design. Dealing with multiple SD images, however, can be cumbersome.
This project describes how to combine multiple designs into a single platform. This will be accomplished with the "xmutil" platform management utility from AMD.
AMD (Xilinx) introduced XMUTIL with the Kria portfolio. The utility can be used to query the platform status, and manage the accelerated apps, as well as several other features.
Of interest to us in this project are the following commands:
- xmutil listapps
- xmutil loadapp {app}
- xmutil unloadapp
These commands call DFX-MGR under the hood, which is AMD (Xilinx)'s implementation of the open source FPGA MANAGER service in linux.
The following table lists which dfx-mgr-client commands are called by the xmutil utility.
It allows the user to dynamically load hardware designs as full bitstreams or as partial reconfiguration, along with the device tree content that describes this hardware.
Creating the new platformThere are many ways to go about combining the "u96v2-sbc-base" and "u96v2-sbc-dualcam" designs into a single "u96v2-sbc" platform.
I have chosen to do this by combining the petalinux project of the "u96v2-sbc-dualcam" design with the Vivado project of the "u96v2-sbc-base" design. The reasons being the following:
- the dualcam petalinux project includes the additional drivers for the MIPI capture pipeline, and is thus a super-set of features that will support both designs
- the base Vivado project has an (almost) empty PL, and is thus easier to remove from the default (empty) device tree content
In order to reuse the petalinux project, we first want to package up the modifications we made (ie. adding jupyter functionality) into a BSP using a meaningful name (ie. u96v2_sbc_dualcam_2022_2_jupyter.bsp), as follows:
$ cd ~/Avnet_2022_2/petalinux/projects
$ petalinux-package --bsp -p u96v2_sbc_dualcam_2022_2 --output u96v2_sbc_dualcam_2022_2_jupyter.bspNext, we want to create a new petalinux project for our new design:
$ cd ~/Avnet_2022_2/petalinux/projects
$ petalinux-create -t project -s u96v2_sbc_dualcam_2022_2_jupyter.bsp -n u96v2_sbc_2022_2We want to rename the design name of our new petalinux project, by modifying the following file:
~/Avnet_2022_2/petalinux/projects/u96v2_sbc_2022_2/project-spec/configs/config
...
#
# Firmware Version Configuration
#
CONFIG_SUBSYSTEM_HOSTNAME="u96v2-sbc-2022-2"
CONFIG_SUBSYSTEM_PRODUCT="u96v2_sbc_2022_2"
CONFIG_SUBSYSTEM_FW_VERSION="1.00"
...
#
# Yocto Settings
#
CONFIG_YOCTO_MACHINE_NAME="u96v2-sbc-base"
...Notice that we set the yocto machine name to "u96v2-sbc-base".
Finally, we configure the petalinux project to use the "base" Vivado project, and attempt our first build:
$ cd ~/Avnet_2022_2/petalinux/projects/u96v2_sbc_2022_2
$ petalinux-config --silentconfig --get-hw-description=../../../hdl/projects/u96v2_sbc_base_2022_2
$ petalinux-buildThis will result in expected errors while building the device tree, related to the MIPI capture pipeline.
Label or path CAPTURE_PIPELINE_mipi_csi2_rx_subsystem_0 not found
Label or path CAPTURE_PIPELINE_v_frmbuf_wr_0 not found
Label or path CAPTURE_PIPELINE_v_proc_ss_scaler_0 not found
Label or path CAPTURE_PIPELINE_v_proc_ss_csc_0 not found
These will be fixed in the next section.
Purging the Device TreeSince we are reusing the petalinux project from the dualcam design, it's device tree failed to build when combined with the base design, which was to be expected.
Since we will be dynamically loading our PL designs, we want to remove all traces of the base and dualcam designs from the default (empty) device tree, and eventually move these to dynamic device tree definitions to each design's firmware overlay.
The first thing to do is configure the project to NOT generate device tree content for the PL. This can be done with the petalinux-config command or in the following file:
~/Avnet_2022_2/petalinux/projects/u96v2_sbc_2022_2/project-spec/configs/config
...
CONFIG_SUBSYSTEM_REMOVE_PL_DTB=y
...For the Ultra96-V2's dualcam design, additional device tree content is defined in the following file:
~/Avnet_2022_2/petalinux/projects/u96v2_sbc_2022_2/project-spec/meta-on-semiconductor/recipes-bsp/device-tree/files/u96v2-sbc-dualcam/system-user.dtsi
Our new petalinux design is defaulting to the dualcam design's device tree, so we can remove this device tree content:
$ rm -r project-spec/meta-on-semiconductor/recipes-bsp/device-treeThere is also device content in the following file:
~/Avnet_2022_2/petalinux/projects/u96v2_sbc_2022_2/project-spec/meta-avnet/recipes-bsp/device-tree/files/u96v2-sbc/system-bsp.dtsi
We want to remove the content that makes reference to peripherals in the PL. This can be done be commenting out the PL definitions (axi_intc_0) as follows:
~/Avnet_2022_2/petalinux/projects/u96v2_sbc_2022_2/project-spec/meta-avnet/recipes-bsp/device-tree/files/u96v2-sbc/system-bsp.dtsi
/* Vitis interrupt connected at PS interrupt offset [6] (89+6=95) */
/* This description may be missing/incorrect in the pl.dtsi, so we add/fix it here */
/*
&axi_intc_0 {
compatible = "xlnx,xps-intc-1.00.a";
interrupt-parent = <&gic>;
interrupts = <0 95 1>;
};
&amba_pl {
zyxclmm_drm {
compatible = "xlnx,zocl";
status = "okay";
interrupt-parent = <&axi_intc_0>;
interrupts = <0 4>, <1 4>, <2 4>, <3 4>,
...
};
};
*/We can rebuilt the petalinux project (again):
$ petalinux-buildNow that we have our petalinux project set up as a "clean" slate, we can add our two (2) designs as firmware overlays, which we will name:
- {vendor}_{platform}_{design}
- avnet_u96v2_base
- avnet_u96v2_dualcam
Petalinux provides a command to create a yocto recipe for these firmware overlays:
$ petalinux-create -t apps
--template fpgamanager -n {firmware}
--enable
--srcuri"{path}/{firmware}.bit
{path}/{firmware}.dtsi
{path}/{firmware}.xclbin
{path}/shell.json"
--forceBefore using this command, however, we need to setup the files required for our firmware.
The bitstream, {firmware}.bit, can be found in the ~/Avnet_2022_2/hdl/projects/ directories.
The dynamic device tree, {firmware}.dtsi, will have to be created from existing content.
The vitis container, {firmware}.xclbin, is not required for these designs, but will be covered in the next project when we add Vitis-AI support to our platform.
The shell.json is a simple xml file describing the overlay type.
We start by copying the.bit and creating the shell.json files for the base design:
$ mkdir -p firmware/avnet_u96v2_base
$ cp ../../../hdl/projects/u96v2_sbc_base_2022_2/u96v2_sbc_base.runs/impl_1/u96v2_sbc_base_wrapper.bit firmware/avnet_u96v2_base/avnet_u96v2_base.bit
$ echo '{ "shell_type":"XRT_FLAT", "num_slots":1 }' > firmware/avnet_u96v2_base/shell.jsonThen we copy the.bit and create the shell.json files for the dualcam design:
$ mkdir -p firmware/avnet_u96v2_dualcam
$ cp ../../../hdl/projects/u96v2_sbc_dualcam_2022_2/u96v2_sbc_dualcam.runs/impl_1/u96v2_sbc_dualcam_wrapper.bit firmware/avnet_u96v2_dualcam/avnet_u96v2_dualcam.bit
$ echo '{ "shell_type":"XRT_FLAT", "num_slots":1 }' > firmware/avnet_u96v2_dualcam/shell.jsonNow we need to create the device tree content (.dtsi) for our designs.
Use the following content as a starting point.
avnet_u96v2_base.dtsi
/dts-v1/;
/plugin/;
#include <dt-bindings/gpio/gpio.h>
&fpga_full {
#address-cells = <2>;
#size-cells = <2>;
firmware-name = "avnet_u96v2_base.bit.bin";
resets = <&zynqmp_reset 116>, <&zynqmp_reset 117>, <&zynqmp_reset 118>, <&zynqmp_reset 119>;
}
&amba {
afi0: afi0 {
compatible = "xlnx,afi-fpga";
config-afi = <0 0>, <1 0>, <2 0>, <3 0>, <4 1>, <5 1>, <6 0>, <7 0>, <8 1>, <9 1>, <10 2>, <11 2>, <12 2>, <13 2>, <14 0x000>, <15 0x000>;
resets = <&zynqmp_reset 116>, <&zynqmp_reset 117>, <&zynqmp_reset 118>, <&zynqmp_reset 119>;
reset-names = "pl0", "pl1", "pl2", "pl3";
};
};
&amba {
/* copy content of u96v2_sbc_base_2022_2/components/plnx_workspace/device-tree/device-tree/pl.dtsi here */
}avnet_u96v2_dualcam.dtsi
...
/dts-v1/;
/plugin/;
#include <dt-bindings/gpio/gpio.h>
&fpga_full {
#address-cells = <2>;
#size-cells = <2>;
firmware-name = "avnet_u96v2_dualcam.bit.bin";
resets = <&zynqmp_reset 116>, <&zynqmp_reset 117>, <&zynqmp_reset 118>, <&zynqmp_reset 119>;
}
&amba {
afi0: afi0 {
compatible = "xlnx,afi-fpga";
config-afi = <0 0>, <1 0>, <2 0>, <3 0>, <4 1>, <5 1>, <6 0>, <7 0>, <8 1>, <9 1>, <10 2>, <11 2>, <12 2>, <13 2>, <14 0x000>, <15 0x000>;
resets = <&zynqmp_reset 116>, <&zynqmp_reset 117>, <&zynqmp_reset 118>, <&zynqmp_reset 119>;
reset-names = "pl0", "pl1", "pl2", "pl3";
};
};
&amba {
/* copy content of u96v2_sbc_dualcam_2022_2/components/plnx_workspace/device-tree/device-tree/pl.dtsi here */
}Now we can copy the contents of the design specific pl.dtsi from the original petalinux projects:
~/Avnet_2022_2/petalinux/projects/u96v2_sbc_base_2022_2/components/plnx_workspace/device-tree/device-tree/pl.dtsi
~/Avnet_2022_2/petalinux/projects/u96v2_sbc_dualcam_2022_2/components/plnx_workspace/device-tree/device-tree/pl.dtsi
For the dualcam overlay, we need to add the additional content for the dualcam. This can also be added at the end of the avnet-u96v2-dualcam.dtsi file:
&amba {
ap1302_osc: ap1302oscillator {
compatible = "fixed-clock";
#clock-cells = <0>;
clock-frequency = <48000000>;
clock-output-names = "ap1302osc";
};
};
...We can now create our firmware overlays, as follows:
$ petalinux-create -t apps \
--template fpgamanager -n avnet-u96v2-base \
--enable \
--srcuri "firmware/avnet_u96v2_base/avnet_u96v2_base.bit \
firmware/avnet_u96v2_base/avnet_u96v2_base.dtsi \
firmware/avnet_u96v2_base/shell.json" \
--force
$ petalinux-create -t apps \
--template fpgamanager -n avnet-u96v2-dualcam \
--enable \
--srcuri "firmware/avnet_u96v2_dualcam/avnet_u96v2_dualcam.bit \
firmware/avnet_u96v2_dualcam/avnet_u96v2_dualcam.dtsi \
firmware/avnet_u96v2_dualcam/shell.json" \
--forceThis will have created new entries in the user-rootfsconfig and rootfs_config configuration files. Add the "xmutil" package to these, as follows:
project-spec/meta-user/conf/user-rootfsconfig
...
CONFIG_avnet-u96v2-base
CONFIG_avnet-u96v2-dualcam
CONFIG_xmutil
...project-spec/configs/rootfs_config
...
#
# apps
#
CONFIG_avnet-u96v2-base=y
CONFIG_avnet-u96v2-dualcam=y
CONFIG_xmutil=y
...Since we now defined our firmware overlays, we want to include the packages that enable the ap1302 kernel driver and demo scripts. This can be done by adding the following at the end of the following file:
project-spec/meta-on-semiconductor/recipes-core/images/petalinux-image-minimal.bb
...
IMAGE_INSTALL:append:u96v2-sbc-base += "\
ap1302 \
libdrm \
libdrm-tests \
libdrm-kms \
dualcam-python-examples \
"Note that we are not including the "camera-setup" package, since this does not apply to our common image. We also want to remove this "camera-setup" package from the "dualcam-python-examples" package, and replace it with "avnet-u96v2-dualcam"
project-spec/meta-on-semiconductor/recipes-app/dualcam-python-examples/dualcam-python-examples.bb
...
RDEPENDS:${PN} = "python3 \
python3-numpy \
opencv \
avnet-u96v2-dualcam \
"
...We can rebuild the petalinux project (yet again):
$ petalinux-buildIn order to verify our petalinux image and base & dualcam overlays, the best way I have found is to execute them.
In order to execute the petalinux image, we first need to program the SD card image to a micro-SD card (of size 32GB or greater).
~/Avnet_2022_2/petalinux/projects/u96v2_sbc_2022_2/images/linux/rootfs.wic
To do this, we use Balena Etcher, which is available for most operating systems.
Once programmed, insert the micro-SD card into the Ultra96-V2, and connect up the platform as shown below.
Make sure the DualCam mezzanine has the jumpers configured as follows:
- J1 => IAS0
- J2 => IAS1
- J3 => 96Boards High-Speed Connector
- J4 => 96Boards Low-Speed Connector
- J5 => MIPI DSI
- P1 – 1-2 => SENSOR1_GPIO1 is +2V8_AF
- P2 – 2-3 => SENSOR2_GPIO1 is 96B_SP3
- P3 – present => SENSOR1_GPIO0/FLASH is GND
- P4 – present => SENSOR2_GPIO0/FLASH is GND
- P5 – 2-3 => SENSOR1_GPIO3 is GND
- P6 – 2-3 => SENSOR2_GPIO3 is SENS2_ADDR
- P7 – 1-2 => PWR_SEL is 5V
- P8-P9 – P8.2-P8.3 => VDD_SEL is 1.2V
- P10 – 1-2 => VAA_SEL is 2.7V (3-2 => 2.8V)
- P11 – 2-3 => CLK_SEL is on-board OSC(48 MHz)
Press the power push-button to boot the board, and login as "root".
As linux boots, a large quantity of verbose output will be sent to the serial console, ending with the following:
...
Starting Network Time Service...
Starting Permit User Sessions...
Starting Target Communication Framework agent...
[ OK ] Started Xinetd A Powerful Replacement For Inetd.
[ OK ] Finished Permit User Sessions.
[ OK ] Started Getty on tty1.
[ OK ] Started Serial Getty on ttyPS0.
[ OK ] Reached target Login Prompts.
[ OK ] Started Target Communication Framework agent.
[ OK ] Started Network Time Service.
Starting Hostname Service...
[ OK ] Started Hostname Service.
[ OK ] Finished Access point for Ultra96.
[ OK ] Reached target Multi-User System.
[ OK ] Reached target Graphical Interface.
Starting Record Runlevel Change in UTMP...
[ OK ] Finished Record Runlevel Change in UTMP.
PetaLinux 2022.2_release_S10071807 u96v2-sbc-2022-2 ttyPS0Login as the "root" user as follows:
u96v2-sbc-2022-2 login: root
root@u96v2-sbc-2022-2:~#If we made it this far, our petalinux project is working. We can now verify pour "avnet_u96v2_base" and "avnet_u96v2_dualcam" overlays, which are called "apps" by xmutil.
We start by querying which "apps" are present:
root@u96v2-sbc-2022-2:~# xmutil listapps
Accelerator Accel_type Base Base_type #slots Active_slot
avnet-u96v2-base XRT_FLAT avnet-u96v2-base XRT_FLAT (0+0) -1
avnet-u96v2-dualcam XRT_FLAT avnet-u96v2-dualcam XRT_FLAT (0+0) -1
root@u96v2-sbc-2022-2:~#Now that we have verified our common u96v2-sbc-2022.2 image, and the presence of our avnet-u96v2-base and avnet-u96v2-dualcam overlays, we can start verifying these overlays.
Verifying these overlays is an iterative process, due to the complexity of the dynamic device tree content.
Verifying the avnet-u96v2-base overlayWe start with the simpler of the two overlays : avnet-u96v2-base.
root@u96v2-sbc-2022-2:~# xmutil loadapp avnet-u96v2-base
[ 57.630122] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/firmware-name
[ 57.640266] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/resets
[ 57.650985] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/afi0
[ 57.660507] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/PWM_w_Int_0
[ 57.670616] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/PWM_w_Int_1
[ 57.680715] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_bram_ctrl_0
[ 57.691164] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_gpio_0
[ 57.701173] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_gpio_1
[ 57.711189] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_gpio_2
[ 57.721197] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_intc_0
[ 57.731213] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_uart16550_0
[ 57.741656] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_uart16550_1
[ 57.752114] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/system_management_wiz_0
[ 59.316501] xadc a0080000.system_management_wiz: IRQ index 0 not found
[ 59.352192] zocl-drm axi:zyxclmm_drm: IRQ index 32 not found
avnet-u96v2-base: loaded to slot 0Note that the following WARNING occurs for the case of a working dynamic device tree, so can be ignored:
OF: overlay: WARNING: memory leak will occur if overlay removedWe can also verify that we can unload the base overlay:
root@u96v2-sbc-2022-2:~# xmutil unloadapp
[ 73.027798] OF: ERROR: memory leak, expected refcount 1 instead of 2, of_node_get()/of_node_put() unbalanced - destroy cset entry: attach overlay node /axi/zyxclmm_drm
[ 73.043639] OF: ERROR: memory leak, expected refcount 1 instead of 225, of_node_get()/of_node_put() unbalanced - destroy cset entry: attach overlay node /axi/interrupt-controller@a0090000
[ 73.060578] OF: ERROR: memory leak, expected refcount 1 instead of 2, of_node_get()/of_node_put() unbalanced - destroy cset entry: attach overlay node /axi/afi0
[ 73.075237] OF: ERROR: memory leak before free overlay changeset, /axi/afi0
remove from slot 0 returns: 0 (Ok)Note that the following WARNING occurs for the case of a working dynamic device tree, so can be ignored:
OF: ERROR: memory leak before free overlay changeset, ...Next, we tackle the more complex of the two overlays : avnet-u96v2-dualcam.
root@u96v2-sbc-2022-2:~# xmutil loadapp avnet-u96v2-dualcam
[ 89.695757] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/firmware-name
[ 89.705895] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/resets
[ 89.715518] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/i2c@ff030000/i2c-mux@75/reset-gpios
[ 89.728401] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/afi0
[ 89.737900] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/ap1302_osc
[ 89.747927] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/ias_out0
[ 89.757772] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[ 89.770303] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/mipi_csi_portsCAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[ 89.784051] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/mipi_csi_port1CAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[ 89.797800] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/mipi_csirx_outCAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[ 89.811549] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/mipi_csi_port0CAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[ 89.825298] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/mipi_csi_inCAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[ 89.838787] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/misc_clk_0
[ 89.848804] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_frmbuf_wr_0
[ 89.860558] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_proc_ss_csc_0
[ 89.872483] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/csc_portsCAPTURE_PIPELINE_v_proc_ss_csc_0
[ 89.885191] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/csc_port1CAPTURE_PIPELINE_v_proc_ss_csc_0
[ 89.897897] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/csc_outCAPTURE_PIPELINE_v_proc_ss_csc_0
[ 89.910432] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/csc_port0CAPTURE_PIPELINE_v_proc_ss_csc_0
[ 89.923140] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_proc_ss_csc_0CAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[ 89.938451] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 89.950637] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/scaler_portsCAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 89.963876] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/scaler_port1CAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 89.977104] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/sca_outCAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 89.989898] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/scaler_port0CAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 90.003127] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_proc_ss_scaler_0CAPTURE_PIPELINE_v_proc_ss_csc_0
[ 90.018090] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/GPIO_axi_gpio_0
[ 90.028540] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_bram_ctrl_0
[ 90.038991] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_gpio_1
[ 90.049006] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_gpio_2
[ 90.059020] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_intc_0
[ 90.069033] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/system_management_wiz_0
[ 90.080178] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/vcap_portsCAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 90.093233] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/vcap_portCAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 90.106201] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_frmbuf_wr_0CAPTURE_PIPELINE_v_proc_ss_scaler_0
[ 90.778858] debugfs: Directory '4-003c' with parent 'regmap' already present!
[ 90.805018] xadc a0080000.system_management_wiz: IRQ index 0 not found
[ 90.815656] zocl-drm axi:zyxclmm_drm: IRQ index 32 not found
avnet-u96v2-dualcam: loaded to slot 0The last verification step is to check the presence of the /dev/media node.
$ ls /dev/media*
/dev/media0
$ media-ctl -p -d /dev/media0There is a good chance that the first attempt did not work, which will require an iterative approach to verification... especially for the device tree content...
Now the fun begins !
It is not required to re-program the SD card image at each iteration. Instead, we can re-build the avnet-u96v2-base or avnet-u96v2-dualcam packages, and re-install them on our image with the dnf utility, as shown below:
$ petalinux-build -c avnet-u96v2-dualcam
$ ls build/tmp/deploy/rpm/u96v2-sbc-base/avnet-u96v2-dualcam-1.0-r0.*.*.rpm
avnet-u96v2-dualcam-1.0-r0.2.u96v2_sbc_base.rpmThis rpm package can be copied over to the embedded platform (via the SD card image, or via SSH), and installed as follows:
root@u96v2-sbc-2022-2:~# dnf install rpm/avnet-u96v2-dualcam-1.0-r0.2.u96v2_sbc_base.rpm
...
Dependencies resolved.
======================================================================================
Package Architecture Version Repository Size
======================================================================================
Upgrading:
avnet-u96v2-dualcam u96v2_sbc_base 1.0-r0.2 @commandline 130 k
Transaction Summary
======================================================================================
Upgrade 1 Package
Total size: 130 k
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Upgrading : avnet-u96v2-dualcam-1.0-r0.2.u96v2_sbc_base 1/2
Cleanup : avnet-u96v2-dualcam-1.0-r0.1.u96v2_sbc_base 2/2
Verifying : avnet-u96v2-dualcam-1.0-r0.2.u96v2_sbc_base 1/2
Verifying : avnet-u96v2-dualcam-1.0-r0.1.u96v2_sbc_base 2/2
Upgraded:
avnet-u96v2-dualcam-1.0-r0.2.u96v2_sbc_base
Complete!The user can configure the image to automatically boot one of the firmware overlays. The /etc/dfx-mgrd/daemon.conf file indicates which overlay (default_accel) to load at boot in the /etc/dfx/mgrd/default_firmware.
root@u96v2-sbc-2022-2:~# cat /etc/dfx-mgrd/daemon.conf
{
"firmware_location": ["/lib/firmware/xilinx"],
"default_accel":"/etc/dfx-mgrd/default_firmware"
}This file does not exist by default, but can be created as follows:
root@u96v2-sbc-2022-2:~# cat /etc/dfx-mgrd/default_firmware
cat: /etc/dfx-mgrd/default_firmware: No such file or directory
root@u96v2-sbc-2022-2:~# echo avnet-u96v2-base > /etc/dfx-mgrd/default_firmware
root@u96v2-sbc-2022-2:~# cat /etc/dfx-mgrd/default_firmware
avnet-u96v2-baseThe change will take effect at the next boot.
root@u96v2-sbc-2022-2:~# rebootAfter boot, we start by querying which "apps" are present:
root@u96v2-sbc-2022-2:~# xmutil listapps
Accelerator Accel_type Base Base_type #slots Active_slot
avnet-u96v2-base XRT_FLAT avnet-u96v2-base XRT_FLAT (0+0) 0,
avnet-u96v2-dualcam XRT_FLAT avnet-u96v2-dualcam XRT_FLAT (0+0) -1
root@u96v2-sbc-2022-2:~#Notice that the avnet-u96v2-base overlay has been loaded.
Executing avnet-u96v2-baseSimilarly to the previous project, we will create a jupyter notebook that performs a USB camera passthrough. We will add code to explicitly load the desired firmware overlay.
We start by launghing the jupyter-lab server in the serial console, specifying the IP address, as follows:
petalinux@u96v2-sbc-dualcam-2022-2:$ jupyter-lab --ip 10.0.0.179 &
[1] 879
[I 2023-03-20 16:07:40.475 ServerApp] jupyterlab | extension was successfully linked.
[I 2023-03-20 16:07:40.548 ServerApp] Writing Jupyter server cookie secret to /home/petalinux/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2023-03-20 16:07:40.649 LabApp] JupyterLab extension loaded from /usr/lib/python3.9/site-packages/jupyterlab
[I 2023-03-20 16:07:40.649 LabApp] JupyterLab application directory is /usr/share/jupyter/lab
[I 2023-03-20 16:07:40.681 ServerApp] jupyterlab | extension was successfully loaded.
[I 2023-03-20 16:07:40.686 ServerApp] Serving notebooks from local directory: /home/root
[I 2023-03-20 16:07:40.686 ServerApp] Jupyter Server 1.13.5 is running at:
[I 2023-03-20 16:07:40.687 ServerApp] http://10.0.0.179:8888/lab?token=06452daa538f08d3ef351ca243735d0a97ac1cca2a95f999
[I 2023-03-20 16:07:40.687 ServerApp] or http://127.0.0.1:8888/lab?token=06452daa538f08d3ef351ca243735d0a97ac1cca2a95f999
[I 2023-03-20 16:07:40.687 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 2023-03-20 16:07:40.717 ServerApp] No web browser found: could not locate runnable browser.
[C 2023-03-20 16:07:40.719 ServerApp]
To access the server, open this file in a browser:
file:///home/petalinux/.local/share/jupyter/runtime/jpserver-879-open.html
Or copy and paste one of these URLs:
http://10.0.0.179:8888/lab?token=06452daa538f08d3ef351ca243735d0a97ac1cca2a95f999
or http://127.0.0.1:8888/lab?token=06452daa538f08d3ef351ca243735d0a97ac1cca2a95f999On the PC side, the jupyter notebook can be accesed by copying the link that contains the token. For example, in my case this was:
http://10.0.0.179:8888/?token=6dae07dd4168bf26850904cadd43526fe146fcc59de4b384
http://10.0.0.179:8888/?token=6dae07dd4168bf26850904cadd43526fe146fcc59de4b384In the "Notebook" section of the "Launcher", click on the "Python3 (ipykernel)" button to create a new jupyter notebook.
We will start by explicitly loading the firmware overlay we want to use : avnet-u96v2-base.
Create code cells and copy-paste the following code snippets:
import subprocess
import re
import osThe following code will query the platform name using the hostname utility.
def get_platform_name():
proc = subprocess.run(['hostname'], capture_output=True, encoding='utf8')
for line in proc.stdout.splitlines():
platform_name = re.search('(.+?)-sbc-(.+?)',line).group(1)
return platform_name
platform_name = get_platform_name()
print(platform_name)The following code will unload the current firmware overlay (if present):
cmd = 'xmutil unloadapp'
print(cmd)
os.system(cmd)The following code will load the base firmware overlay.
cmd = 'xmutil loadapp avnet-'+platform_name+'-base'
print(cmd)
os.system(cmd)Finally, we can paste the same code as in the previous project, to perform the live USB camera feed in the jupyter notebook:
import matplotlib.pyplot as plt
import cv2
import numpy as np
from IPython.display import display, Image
import ipywidgets as widgets
import threading
# Stop button
# ================
stopButton = widgets.ToggleButton(
value=False,
description='Stop',
disabled=False,
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # (FontAwesome names without the `fa-` prefix)
)
# Display function
# ================
def view(button):
cap = cv2.VideoCapture(0)
display_handle=display(None, display_id=True)
i = 0
while True:
_, frame = cap.read()
frame = cv2.flip(frame, 1) # if your camera reverses your image
_, frame = cv2.imencode('.jpeg', frame)
display_handle.update(Image(data=frame.tobytes()))
if stopButton.value==True:
cap.release()
display_handle.update(None)
# Run
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()Save the jupyter notebook with a meaningful name (ie. usbcam_live_view.ipynb), then click the "Run" icon on each of the two code cells.
You will see a live video feed in the jupyter notebook, as shown below:
Click the "Stop" widget to stop the live feed.
Known IssuesThe current version of this project has the following known issues:
- avnet-u96v2-dualcam sometimes does not stream video
I'm working on it...
ConclusionI hope this tutorial helped to understand the benefits of using "xmutil" and the dynamic configuration of the PL to create a single SD image for your custom AI applications on the Ultra96-V2.
If you would like to have the pre-built petalinux BSP or SDcard image for this common platform, please let me know in the comments below.
Revision History2023/03/27
Preliminary Version
Comments
Please log in or sign up to comment.