Implementation DPU, GPIO, and PWM for KR260

Using Vivado and Vitis, we created a project to synthesize the DPU IP.

We utilized the DPU created on PYNQ with KR260 to perform object detection using Vitis AI (Yolo).

In this post, we will introduce the process of running GPIO (PWM) alongside the DPU on KR260.

Subproject for the AMD Pervasive AI Developer Contest

This project is part of a subproject for the AMD Pervasive AI Developer Contest.

Be sure to check out the other projects as well.

***The main project is currently under submission. ***

0. Main project << under submission

1. PYNQ + GPIO(LED Blinking)

2. PYNQ + PWM(DC-Motor Control)

3. Object Detection(Yolo) with DPU-PYNQ

4. Implementation DPU, GPIO, and PWM << this project

5. Remote Control 360° Camera

6. GStreamer + OpenCV with 360°Camera

7. 360 Live Streaming + Object Detect(DPU)

8. ROS2 3D Marker from 360 Live Streaming

9. Control 360° Object Detection Robot Car

10. Improve Object Detection Speed with YOLOX

11. Benchmark Architectures of the DPU

12. Power Consumption of 360° Object Detection Robot Car

13. Application to Vitis AI ONNX Runtime Engine (VOE)

14. Appendix: Object Detection Using YOLOX with a Webcam

Please note that before running the above subprojects, the following setup, which is the reference for this AMDcontest, is required.

https://github.com/amd/Kria-RoboticsAI

Introduction

Using Vivado and Vitis, we created a project to synthesize the DPU IP.

This project utilizes the DPU- PYNQ for the KR260. We performed object detection using Vitis AI (Yolo).

We will also introduce the process of running GPIO (PWM) alongside the DPU.

Below is a test video summarizing our work.

Creation Process

There are various methods to create a project that includes the DPU IP.

This example follows a specific process.

The author created this project in the Vivado 2023.1 and Vitis 2023.1 environment, so adjust accordingly if following along.

• Synthesize the MPSoC, clock, and reset required for the DPU using Vivado.
• Synthesize the DPU using Vitis.
• Synthesize other IPs, such as GPIO and PWM, using Vivado.

Creating the DPU Project in Vivado

The.tcl file for the Vivado project is located in the following directory.

https://github.com/iotengineer22/AMD-Pervasive-AI-Developer-Contest/tree/main/jupyter_notebooks/pynq-original-dpu-model/vivado

If you want to create it automatically using the Tcl file, use the following command:

vivado -mode batch -source amd_contest_design.tcl

However, you will need to manually perform the process from the "Export Platform" step.

Create Project

Open Vivado and start creating the project. First, place the MPSoC, clock, and reset IPs required for the DPU.

Select "Create Project" and proceed with an appropriate name.

For Project Type, use the default RTL.

For Default Part, select the board and choose KR260.

Create Block Design

Create a Block Design from the project file.

From the "+" icon in the Diagram, select Zynq UltraScale+ MPSoC.

Run Block Automation (default settings are OK).

Search for "clock" from the "+" icon in the Diagram and select the Clocking Wizard IP.

Set the clock to output 6 signals (100, 150, 200, 300, 400, 600) and set the reset to Active Low.

Prepare 5 Processor System Reset IPs, matching the number of clocks.

Use Run Block Automation to connect each reset IP to a clock source.

After automatic connections, use "Regenerate Layout" to clean up the IP arrangement.

Further Configuration

Enable "Project is an extensible Vitis platform" in Settings under General.

Open Platform Setup from the Window tab.

Enable all clocks and assign IDs starting from 0, with a default of 300 MHz.

Configure the "PS-PL Configuration" section of the Zynq UltraScale+ MPSoC IP, enabling AXI HPHM0 LPD.

Place an AXI Interrupt Controller IP, setting Interrupt Action to "Single."

Run Block Automation to connect the IPs, then connect the interrupt output to the MPSoC input.

Check "Intr" and configure the AXI Port in Platform Setup, naming up to SPTag.

Generate Bitstream

With this, the creation of the necessary clocks and resets for the DPU is complete.

Right-click the design source and create an HDL wrapper.

Click "Generate Bitstream" to synthesize the design.

At this point, the FPGA utilization is as follows. Since we have not yet integrated the DPU, GPIO, or PWM, the utilization is minimal.

Export Platform

After synthesis, export the platform for Vitis.

Select "Hardware and Hardware emulation" and check "include bitstream."

Name the platform and output the XSA file, typically in the project folder.

Integrating DPU in Vitis

Download and extract the necessary images and IP for the DPU in Vitis.

ZYNQMP Common Image

Download the "ZYNQMP common image" from the provided link, which is a platform combining FPGA and CPU.

https://www.xilinx.com/support/download/index.html/content/xilinx/en/downloadNav/embedded-platforms/2023-1.html

Extract and install the image.

tar -xzvf xilinx-zynqmp-common-v2023.1_05080224.tar.gz
cd xilinx-zynqmp-common-v2023.1/
./sdk.sh -d .

Vitis AI Runtime

Download the Vitis AI Runtime package and extract it into the ZynqMP directory.

wget https://www.xilinx.com/bin/public/openDownload?filename=vitis_ai_2023.1-r3.5.0.tar.gz -O vitis_ai_2023.1-r3.5.0.tar.gz
rm -r xilinx-zynqmp-common-v2023.1/sysroots/cortexa72-cortexa53-xilinx-linux/usr/share/cmake/XRT/
tar -xzvf vitis_ai_2023.1-r3.5.0.tar.gz -C xilinx-zynqmp-common-v2023.1/sysroots/cortexa72-cortexa53-xilinx-linux/

DPUCZDX8G

Download and extract the DPUCZDX8G IP for KR260.

wget https://www.xilinx.com/bin/public/openDownload?filename=DPUCZDX8G.tar.gz -O DPUCZDX8G.tar.gz
tar -xzvf DPUCZDX8G.tar.gz

Register the DPU IP in Vitis by selecting Preferences under the Windows tab and adding the DPUCZDX8G directory in Library Repositories.

Create Platform in Vitis

Create a new platform project in Vitis using the.xsa file created in Vivado, selecting Linux as the OS.

Build the platform by selecting platform.spr and clicking the build icon.

Create Project in Vitis

Create a new application project in Vitis, selecting the previously created platform.

Name the application and specify the path to the ZYNQMP sysroot.

Select the pre-registered DPU Kernel.

Configure the application, setting Active build to "Hardware" and reducing the number of DPU IPs if needed.(2>>1)

Modify dpu_conf.vh in the src/prj/Vitis directory, using the provided PYNQ-DPU settings as references.

I modified the default settings, including URAM and DWCV configurations. I used the following PYNQ-DPU settings as a reference and utilized the B4096 size for the DPU.

https://qiita.com/basaro_k/items/dc439ffbc3ea3aed5eb2

https://github.com/Xilinx/DPU-PYNQ/blob/master/boards/README.md#dpu-ip

`define B4096 
`define URAM_ENABLE 
//config URAM
`ifdef URAM_ENABLE
    `define def_UBANK_IMG_N          6
    `define def_UBANK_WGT_N          17
    `define def_UBANK_BIAS           1
`elsif URAM_DISABLE
    `define def_UBANK_IMG_N          0
    `define def_UBANK_WGT_N          0
    `define def_UBANK_BIAS           0
`endif
`define DRAM_DISABLE 
//config DRAM
`ifdef DRAM_ENABLE
    `define def_DBANK_IMG_N          1 
    `define def_DBANK_WGT_N          1
    `define def_DBANK_BIAS           1
`elsif DRAM_DISABLE
    `define def_DBANK_IMG_N          0
    `define def_DBANK_WGT_N          0
    `define def_DBANK_BIAS           0
`endif
`define RAM_USAGE_LOW
`define CHANNEL_AUGMENTATION_ENABLE
`define DWCV_ENABLE
`define ALU_PARALLEL_DEFAULT 
`define CONV_RELU_LEAKYRELU_RELU6
`define ALU_RELU_RELU6
`define DSP48_USAGE_HIGH 
`define LOWPOWER_DISABLE
`define MPSOC

Build the DPU

Set up the clock and AXI connections for the DPU, modifying V++ settings to disable unnecessary clocks and connections. Therefore, I commented out the unnecessary second sections with #.

[clock]
freqHz=300000000:DPUCZDX8G_1.aclk
freqHz=600000000:DPUCZDX8G_1.ap_clk_2
#freqHz=300000000:DPUCZDX8G_2.aclk
#freqHz=600000000:DPUCZDX8G_2.ap_clk_2
[connectivity]
sp=DPUCZDX8G_1.M_AXI_GP0:HPC0
sp=DPUCZDX8G_1.M_AXI_HP0:HP0
sp=DPUCZDX8G_1.M_AXI_HP2:HP1
#sp=DPUCZDX8G_2.M_AXI_GP0:HPC0
#sp=DPUCZDX8G_2.M_AXI_HP0:HP2
#sp=DPUCZDX8G_2.M_AXI_HP2:HP3

Finally, build the project. Select the..._hw_link[pl] section and click the build icon at the top. Depending on your PC's specifications, this process may take several minutes to a few hours.

After completing the build in Vitis, verify the model with the integrated DPU.

The model can be found deep within the Hardware directory of the Vitis project (***_hw_link), as shown in the example below:

~/***_hw_link/Hardware/dpu.build/link/vivado/vpl/prj

When you open the project in Vivado, you can see the DPU IP as shown below.

Including the DPU significantly increases the FPGA utilization.

Add GPIO and PWM IPs in Vivado

After building the DPU, add GPIO and PWM IPs in Vivado.

This time, we added 2 GPIO IPs and 4 PWM IPs.

Create an HDL Wrapper and set up XDC files for pin assignments, tailored to the KR260's PMOD connectors.

#PMOD1 motor-driver1
set_property PACKAGE_PIN H12 [get_ports PWM_0]
set_property PACKAGE_PIN E10 [get_ports PWM_1]
set_property PACKAGE_PIN D10 [get_ports PWM_2]
set_property PACKAGE_PIN C11 [get_ports PWM_3]
set_property PACKAGE_PIN B10 [get_ports gpio_rtl_0_tri_o[0]]

#PMOD2 motor-driver2
set_property PACKAGE_PIN J11 [get_ports gpio_rtl_0_tri_o[5]]
set_property PACKAGE_PIN J10 [get_ports gpio_rtl_0_tri_o[6]]
set_property PACKAGE_PIN H11 [get_ports gpio_rtl_0_tri_o[1]]

#PMOD3 infrared sensor
set_property PACKAGE_PIN AE12 [get_ports gpio_rtl_1_tri_i[0]]
set_property PACKAGE_PIN AF12 [get_ports gpio_rtl_1_tri_i[1]]

#PMOD4 debug(led-sw)
set_property PACKAGE_PIN AC12 [get_ports gpio_rtl_0_tri_o[2]]
set_property PACKAGE_PIN AD12 [get_ports gpio_rtl_0_tri_o[3]]
set_property PACKAGE_PIN AE10 [get_ports gpio_rtl_0_tri_o[4]]
set_property PACKAGE_PIN AF10 [get_ports gpio_rtl_1_tri_i[2]]


set_property IOSTANDARD LVCMOS33 [get_ports PWM_0]
set_property IOSTANDARD LVCMOS33 [get_ports PWM_1]
set_property IOSTANDARD LVCMOS33 [get_ports PWM_2]
set_property IOSTANDARD LVCMOS33 [get_ports PWM_3]
set_property IOSTANDARD LVCMOS33 [get_ports gpio_rtl_0_tri_o[0]]
set_property IOSTANDARD LVCMOS33 [get_ports gpio_rtl_0_tri_o[1]]
set_property IOSTANDARD LVCMOS33 [get_ports gpio_rtl_0_tri_o[2]]
set_property IOSTANDARD LVCMOS33 [get_ports gpio_rtl_0_tri_o[3]]
set_property IOSTANDARD LVCMOS33 [get_ports gpio_rtl_0_tri_o[4]]
set_property IOSTANDARD LVCMOS33 [get_ports gpio_rtl_0_tri_o[5]]
set_property IOSTANDARD LVCMOS33 [get_ports gpio_rtl_0_tri_o[6]]
set_property IOSTANDARD LVCMOS33 [get_ports gpio_rtl_1_tri_i[0]]
set_property IOSTANDARD LVCMOS33 [get_ports gpio_rtl_1_tri_i[1]]
set_property IOSTANDARD LVCMOS33 [get_ports gpio_rtl_1_tri_i[2]]

Generate the bitstream with all IPs included.

Running the DPU on KR260 with PYNQ

To run the DPU on PYNQ with KR260, you need the following three files: .bit, .xclbin, and .hwh.

These files are located within the Vitis project (***_hw_link).

Copy and rename them for transfer to KR260. I named them "dpu.***".

• dpu.bit → ~/***_hw_link/Hardware/dpu.build/link/vivado/vpl/prj/prj.runs/impl_1
• dpu.xclbin → ~/***_hw_link/Hardware
• dpu.hwh → ~/***_hw_link/Hardware/dpu.build/link/vivado/vpl/prj/prj.gen/sources_1/bd/design_1/hw_handoff

Transfer the necessary files (.bit,.xclbin,.hwh) to KR260.

Below is an example of copying to KR260 at 192.168.11.7:

scp -r pynq-original-dpu-model/ ubuntu@192.168.11.7:/home/ubuntu/

Jupyter Notebook

Within the /root/jupyter_notebooks/ directory in KR260, a folder is created to house the executed .ipynb file alongside the .bit and .hwl files produced by Vivado.

Below is an example of copying to the jupyter_notebooks directory on the KR260.

sudo su
cd $PYNQ_JUPYTER_NOTEBOOKS
cd jupyter_notebooks/
ls
cp -rf /home/ubuntu/pynq-original-dpu-model/ ./

After KR260 installation, the IP address is confirmed using ifconfig; in my case, it was 192.168.11.9.

Use the Kria-PYNQ environment via Jupyter Notebook to control the PWM. Connect to the KR260 board using a LAN cable and find the IP address using ifconfig. Access the Jupyter Notebook at http://<IP_ADDRESS>:9090/.

ipynb File

The test .bit,.hwh,.xclbin, and.ipynb(my-dpu-gpio-test.ipynb) files are available on GitHub.

And we have provided the .xmodel(kr260_yolov3_tf2.xmodel) at the following link. This is the model used for testing the DPU (object detection).

https://github.com/iotengineer22/AMD-Pervasive-AI-Developer-Contest/tree/main/jupyter_notebooks/pynq-original-dpu-model

Run the modified Jupyter notebook on PYNQ, which includes additional configurations for GPIO and PWM.

The main modifications include applying the DPU overlay to a standard overlay and configuring GPIO(PWM) outputs based on detected objects.

Below are the key points of this program.

from pynq_dpu import DpuOverlay
overlay = DpuOverlay("/root/jupyter_notebooks/pynq-original-dpu-model/dpu.bit")

from pynq import Overlay
from pynq.lib import AxiGPIO
ol = overlay

# LED(GPIO)_set
gpio_0_ip = ol.ip_dict['axi_gpio_0']
gpio_out = AxiGPIO(gpio_0_ip).channel1
mask = 0xffffffff

Test PCB

The data for the LED PCB for KR260 used in this test for GPIO and PWM is available on the following GitHub:

GitHub Repository for PCB KR260 PMOD Test

Test Results

Here is the test video demonstrating the setup.

We have successfully performed object detection (Yolo) using the DPU.

Additionally, we have managed to integrate and operate GPIO (PWM) alongside the DPU.

If the specified objects (Bus, Ball) are detected in the photos within the img folder, specific GPIO and PWM outputs(LED) are triggered.

Reference

Many thanks for reference articles.

https://qiita.com/lp6m/items/df1b87b11f8275ee6210

https://qiita.com/basaro_k/items/dc439ffbc3ea3aed5eb2

https://github.com/Xilinx/DPU-PYNQ/blob/master/boards/README.md#dpu-ip

https://github.com/Xilinx/Vitis-Tutorials/tree/2023.1/Vitis_Platform_Creation/Design_Tutorials/01-Edge-KV260

Conclusion

We synthesized the DPU, GPIO, and PWM IPs for KR260, successfully performing object detection and controlling GPIO and PWM outputs.

Next, we plan to operate a 360° camera (RICOH THETA) with KR260.

Remote Control 360° Camera << next project