Introduction
Auto Generation of Deep Neural Network IP procedure
Simulation/Hardware in the loop simulation (HIL
Results
Remarks

Team Automation/University of Miskolc, Hungary:

•

Published March 31, 2022 © GPL3+

Autonomous robot control modelling with DNN IP generator

Have you ever try to accelerate your AI-based process? The tool we propose is able to generate a DNN IP to deploy on a dedicated hardware

AdvancedProtip1.75 hours238

Autonomous robot control modelling with DNN IP generator

Things used in this project

Hardware components

AMD Kria KV260 Vision AI Starter Kit

Software apps and online services

AMD Vitis Unified Software Platform

Use Vitis 2021.1.1 with Apache Log4j Vulnerability patch

AMD Vivado Design Suite

Use Vivado 2021.1.1 with Apache Log4j Vulnerability patch

MATLAB

Use R2021a

Story

Introduction

Our target is to create an autonomous indoor robot, which use different control strategies to handle its movements. In this project we present one solution, which model the sensors and implement the model predictive control (MPC).

Some word about MPC control: The model predictive control (MPC) is a well-known control strategy used to solve online optimization problems, and is widely used in autonomous vehicle applications. MPC is computationally expensive and resource-consuming, especially when it comes to the limited resources of embedded computing platforms such as system-on-chip (SoC) and field programmable gate array (FPGA) platforms. Recently, machine learning (ML) has become an effective alternative to classical control systems in terms of reducing the execution time and resource consumption. Model predictive control (MPC) is one of the most commonly used control strategies due to its ability to solve an online optimization problem and handle soft and hard constraints. However, for complex and high non-linear systems, the implementation of MPC is a great challenge; it is often infeasible due to its high computational demands, especially with resource-limited embedded platforms.

The deep neural network is considered a self-optimized method due to its ability to optimize its behavior based on the provided information, and that makes DNNs suitable for complex dynamic systems. Using Matlab Simulink and the MPC tool we found that this creates headache is we use small FPGAs. For this reason at University of Miskolc, we created a DNN tool, which creates from the given input data a parametrizable DNN network based on the System Generator library elements from the given data.

A traditional MPC controller is presented in Fig. 1. The prediction strategy of the MPC controller is performed over the prediction horizon, which represents the next P time steps that the controller looks forward to the future. The MPC controller simulates several future scenarios, and the optimizer chooses the best scenario based on the cost function, which represents the error between the reference target and the predicted out- puts. The optimal scenario corresponds to the minimum cost function.

Fig. 1 Traditional MPC diagram.

The controller is computationally expensive since it solves the online optimization problems every time step, which requires high computing capabilities and large memory. The computational load and the high resource consumption make the implementations of MPC on limited resources a big challenge.

The first step in designing the MPC model is to determine the input-output signals of the internal vehicle model and the second step is to set the parameters and determine the constraints. The manipulated variable (steering angle δ) and the disturbance (vx ρ) are determined as input signals, while lateral velocity vy, lateral deviation d, yaw angle θ, and yaw rate ω are determined as output signals (See Fig. 2).

Fig.2 Global position of the vehicle model.

The model equations are as follows:

Fig. 3 Vehicle equations.

The constraints are determined as follows: the steering angle is in the range [-1.04, 1.04] rad and the yaw angle rate is in the range [-0.26, 0.26] rad. The parameters were maintained during the design process until satisfactory behavior was obtained. The overall design of the MPC and plant model is shown in Fig. 4.

Fig.4 General diagram of the MPC and plant model design.

Auto Generation of Deep Neural Network IP procedure

The design steps are as follow:

First the traditional of the MPC controller was made; Validate and test the MPC design ; Design of the Deep Neural Network Model, Generate the data sets; Train the DNN model; Validate and test the DNN model; (These steps are particular for each new DNN so these steps are not presented here);
The next step is to run the IP tool generator (see tool in the attachment);
Run the simulation and hardware in the loop simulation (see the model and the init file attached);
FPGA Deployment and test in hardware.
Create the adaptive hardware platform for KRIA SOM KV260 (see tutorial https://github.com/Xilinx/Vitis-Tutorials/tree/2021.2/Vitis_Platform_Creation/Design_Tutorials/01-Edge-KV260)
Import the AXI IP DNN to Vivado or Vitis (at the moment of editing and submitting the project we did not get feedback from Xilinx how to import in Vitis the Translated project)

For the auto generation of deep neural network IP the created DNN tool (See attachment file DNN Tool IP generator).

First with the DNN tool the DNN was created. In general terms one who will use the tool should give the following parameters:

The inputs of the tool are the following:

the number of neurons for each layer
the precision (floating point or FIX16 or FIX8)
the weights
biases

The output is a DNN IP generated in Matlab Simulink.

Remark: the tool is working but some optimizations and embellishment should be considered for different AMD Xilinx platforms. This is a function of the number of neurons and DSP elements contained by the PL (programmable logic). The tool was tested successfully for KC705 board and for Kria KV260 SOM.

First load the DNN_Generator.m file and edit as needed:

%Edit model_parameters_here...neuron_count = [6 20 10 10 1]; % number of neurons in each of the 7 layers%starting model...init;

Then run. The model is created and can be used for own purposes.

Simulation/Hardware in the loop simulation (HIL)

Simulation was done in the Simulink environment and we obtained good results. Unfortunately for the Kria KV260 module the System Generator did not permit us to do Hardware in the loop simulation. We made HIL with the KC705 board for the DNN generated. The results obtained with the on KV260 simulation and HIL with KC705 were similar.

The Vivado project created as presented in the following figure (Fig. 5) was translated successfully and exported to Vitis. In the Vitis software we were unable to run even the "hello world" project. Vitis reported "platform not recognized.

Fig. 5. Vivado Block Design view.

In the figure (Fig. 5) can be observed that we created two IP-s one which is AXI compatible and one which has no AXI interface. Our experience was that one without AXI interface worked more correctly on the KC705 platform.

Just to show an example in the following figures we present the results for the the KC705 module, since we could not do HIL and tests on KV260.

Fig. 6. Estimated steering angle computed on FPGA compared to simulation results of the traditional MPC and the DNN model.

Fig. 7. Vehicle response in terms of lateral deviation to the estimated steering angle: comparison of FPGA and simulation results.

Fig. 8. Vehicle response in terms of yaw angle to the estimated steering angle: comparison of FPGA and simulation results.

Results

The DNN_tool created is useful to implement any DNN network and can be tested in Simulink. Simulation, HIL and implementation can be done in FGPA.

The created tool is free to be used by anyone.

Unfortunately we were not able to test the Model Predictive Control in the Kria board, but probably in the near future we will solve the problem with the help of AMD-Xilinx staff.

The main problem we faced was how to integrate the DNN MPC model to Vitis, without using RTL kernel Wizard.

Remarks

How long it takes (for a person with advanced knowledge) to build a project?

The answer is: 1 hour to train and prepare the DNN. 5 minutes to install the DNN-tool and generate the IP. 10-20 minutes to create a Vivado project by importing the IPs and doing all the necessary setup. 5-30 minutes (depending on the size of the DNN and the optimization strategy) to create the Bitstream. 10 minutes to migrate to Vitis and run the project. 30 - 60 minutes to write the C program and test the results.

How long took us to build the DNN tool and test the tool on different platforms?

The answer is: 5 month working every day around 3-5 hours.

Code

#include <stdio.h>
#include <stdint.h>

#include "mb_interface.h"
#include "xparameters.h"
#include "xtmrctr.h"
#include "xuartlite.h"
#include "xintc.h"
#include "xgpio.h"

enum { nuof_data= 8 };
enum { DD_PERIOD= 100 };

uint32_t data[nuof_data];
int dptr= 0;
int bptr= 0;
volatile int timer_cyc= 0;

void timer_handler(void *CallBackRef, u8 TmrCtrNumber);
void uart_handler(void *CallBackRef, unsigned int ByteCount);

XIntc iintc;
XUartLite iuart;
XTmrCtr itimer;
XGpio igpio;

void
ddinit_intc()
{
	XIntc_Initialize(&iintc, 0);
	XIntc_Connect(&iintc, 0, (XInterruptHandler)&XTmrCtr_InterruptHandler, &itimer);
	XIntc_Connect(&iintc, 1, (XInterruptHandler)&XUartLite_InterruptHandler, &iuart);
	XIntc_Enable(&iintc, 0);
	XIntc_Enable(&iintc, 1);
	XIntc_Start(&iintc, XIN_REAL_MODE);
}

void
ddinit_uart()
{
	XUartLite_Initialize(&iuart, 0);
	XUartLite_SetSendHandler(&iuart, &uart_handler, NULL);
	XUartLite_EnableInterrupt(&iuart);
}

void
ddinit_gpio()
{
	XGpio_Initialize(&igpio, 0);
	XGpio_SetDataDirection(&igpio, 1, 0x00000000);
	XGpio_SetDataDirection(&igpio, 2, 0x00000000);
}

uint32_t rl;

void
ddinit_timer()
{
	//XPAR_AXI_TIMER_0_CLOCK_FREQ_HZ
	XTmrCtr_Initialize(&itimer, 0);
	XTmrCtr_Stop(&itimer, 0);
	XTmrCtr_Stop(&itimer, 1);
	rl= DD_PERIOD * (XPAR_AXI_TIMER_0_CLOCK_FREQ_HZ/1000);
	XTmrCtr_SetResetValue(&itimer, 0, rl);
	XTmrCtr_SetOptions(&itimer, 0,
			XTC_DOWN_COUNT_OPTION|
			XTC_INT_MODE_OPTION|
			XTC_AUTO_RELOAD_OPTION);
	XTmrCtr_SetHandler(&itimer, &timer_handler, NULL);
}

void
dd_init()
{
	ddinit_gpio();
	ddinit_intc();
	ddinit_uart();
	ddinit_timer();
	XTmrCtr_Start(&itimer, 0);
	microblaze_enable_interrupts();
}


void
uart_handler(void *CallBackRef, unsigned int ByteCount)
{
	// nothing
	XGpio_DiscreteWrite(&igpio, 2, timer_cyc&0xff);
}

void
start_send()
{
	XUartLite_Send(&iuart, (u8*)data, nuof_data*sizeof(uint32_t));
}

//! Byte swap unsigned int
uint32_t
swap_uint32(uint32_t val)
{
    val = ((val << 8) & 0xFF00FF00 ) | ((val >> 8) & 0xFF00FF );
    return (val << 16) | (val >> 16);
}

void
read_data()
{
	int i;
	data[0]= 0;
	data[1]= *((volatile uint32_t*)XPAR_DNNMPCAXI_0_DNNMPCAXI_KIMENETD_S_AXI_BASEADDR);
	data[2]= *((volatile uint32_t*)XPAR_DNNMPCAXI_0_DNNMPCAXI_KIMENETOMEGA_S_AXI_BASEADDR);
	data[3]= *((volatile uint32_t*)XPAR_DNNMPCAXI_0_DNNMPCAXI_KIMENETRHO_S_AXI_BASEADDR);
	data[4]= *((volatile uint32_t*)XPAR_DNNMPCAXI_0_DNNMPCAXI_KIMENETTHETA_S_AXI_BASEADDR);
	data[5]= *((volatile uint32_t*)XPAR_DNNMPCAXI_0_DNNMPCAXI_KIMENETU_S_AXI_BASEADDR);
	data[6]= *((volatile uint32_t*)XPAR_DNNMPCAXI_0_DNNMPCAXI_KIMENETVY_S_AXI_BASEADDR);
	data[7]= 0x55aa1122;
	for (i=0;i<8;i++)
		data[i]= swap_uint32(data[i]);
}


void
timer_handler(void *CallBackRef, u8 TmrCtrNumber)
{
	timer_cyc++;
	if (timer_cyc == 2)
	{
		XGpio_DiscreteWrite(&igpio, 1, 1);
		//return;
	}
	if (timer_cyc >= 2)
		read_data();
	dptr= 0;
	bptr= 0;
	start_send();
}


void
dd_loop()
{
	while (1)
	{
		mbar(16);
		__asm__ __volatile__ ( "nop" );
	}
}

void
dd()
{
	dd_init();
	dd_loop();
}