Running Line detection app on Kria KV260(Xilinx DPU)

Plan of record:

1) Take a TensorFlow model: pretrained

Found something here for the beginning: https://github.com/aliyun/conditional-lane-detection

• Found something here for the beginning: https://github.com/aliyun/conditional-lane-detection

2) Integrate this new model with the smart camera app

Tutorial: https://www.hackster.io/hdcoe/running-yolov2-tiny-on-kv260-28f801

• Tutorial: https://www.hackster.io/hdcoe/running-yolov2-tiny-on-kv260-28f801

3) Run the app on an example

Add a euro-truck simulator on the laptop and use the neural network to generate the steer and the speed: https://github.com/marsauto/europilot

• Add a euro-truck simulator on the laptop and use the neural network to generate the steer and the speed: https://github.com/marsauto/europilot

5) Optimise the flow:

Try to configure the DPU to run better

How to use a PyTorch model

1) BeforeStarting

• choose your pre-trained model ( in this case CurveLane - Medium(ResNet-34))
• create your own script using resnet18_quant.py
• use a validation image test set (Robot Car Dataset)

2) Quantization

The computation process in machine learning is expensive so to achieve better performance at the runtime, the model can be optimised by using quantization and pruning. According to Vitis-Ai documentation:

Generally, 32-bit floating-point weights and activation values are used when training neural networks. By converting the 32-bit floating-point weights and activations to 8-bit integer (INT8) format, the Vitis AI quantizer can reduce computing complexity without losing prediction accuracy.

Install requirement for vai_q_pytorch:

a) install python 3.7. If you build from sources, enable the shared libraries.

./configure --enable-shared

3) Compiling the model

4) Configure the smart camera app

Resources:

https://www.xilinx.com/support/documentation/sw_manuals/vitis_ai/1_4/ug1414-vitis-ai.pdf

https://www.hackster.io/hdcoe/running-yolov2-tiny-on-kv260-28f801

https://www.hackster.io/dsp2/create-a-custom-kria-kv260-accelerated-ml-application-54dca7

https://www.xilinx.com/support/documentation/sw_manuals/xilinx2021_2/ug1144-petalinux-tools-reference-guide.pdf

Xiling vai_q_pytorch custom:

https://github.com/Xilinx/Vitis-AI/blob/master/tools/Vitis-AI-Quantizer/vai_q_pytorch/example/resnet18_quant_custom_op.py

Zynq custom:

https://www.xilinx.com/support/documentation/ip_documentation/dpu/v3_2/pg338-dpu.pdf

Validation data set:https://robotcar-dataset.robots.ox.ac.uk