Depth estimation with AI and stereo vision on the KR260

Objectives

Depth estimation from stereo images has been used for autonomous navigation and SLAM (Simultaneous Localization and Mapping) where an autonomous vehicle recognizes its surroundings in 3D as it moves and builds a 3D map.

Calculating depth from pairs of images is not a simple task and a number of traditional algorithms exist, each of them with its pros and its cons. The key is to find the disparity or relative shift of each object on each image. That implies accurately identifying each scene object in both images. Neural Networks excel in capabilities, accuracy and speed at object detection and recognition in images and are a good candidate to perform depth estimation. A public neural network (FadNET, Fast and Accurate Network for Disparity Estimation) exists and is available for Vitis AI therefore it can be the base for a NN-based SLAM engine.

Linux host setup

A Linux VM is setup with Ubuntu 20.04 LTS that supports Vitis 2023.2 that is in turn supported by the latest VitisAI 3.5

Vitis AI setup

Vitis AI is cloned from the repository https://github.com/Xilinx/Vitis-AI following the insttructions in the README and that will make a directory Vitis-AI under home (~/Vitis-AI)

DPU IP setup

The DPU IP for the KR260 is downloaded from here: https://www.xilinx.com/bin/public/openDownload?filename=DPUCZDX8G.tar.gz

The file is extracted ~/Documents to later be included in Vivado

Vivado project

A vivado project is started based on the KR260 board. connections are specified as below:

The IP is added to Vivado catalog:

The tcl script is run to build the block diagram.

The DPU is set up as below.

And after synthesis and implementation the chip usage is fairly modest

Once built, the hardware is exported as xsa file.

Setting the KR260 SD card image

One option for the KR260 is to install Ubuntu instead of Petalinux. These are the instructions to build the SD card with Ubuntu Server 22.04 LTS
https://www.amd.com/en/products/system-on-modules/kria/k26/kr260-robotics-starter-kit/getting-started/setting-up-the-sd-card-image.html

After the SD card is built and the board is booted, it will ask for a new password and to be updated.

Snap is installed on the target with:

sudo snap install xlnx-config --classic --channel=2.x

Install ROS2 on target

ROS2 (Robotic Operating System) is actually not an Operating System but a framework to work with robotics applications. It can be installed on the KR260 on top of Ubuntu by following these directions:
https://xilinx.github.io/kria-apps-docs/kr260/build/html/docs/ros2_perception_node/docs/app_deployment.html

Get FadNET

FadNET is one of the NNs in the AMD/ Xilinx model zoo (https://xilinx.github.io/Vitis-AI/3.0/html/docs/workflow-model-zoo.html) that can be obtained from github (https://github.com/Xilinx/Vitis-AI/tree/v3.5/examples/vai_library)

A number of variants of FadNET are included in the zoo library. These are already compiled and there is no need to train or optimize.