AXI lite slave custom IP-Core to control Led under petalinux

This blog aims to illustrate the development workflow for Xilinx SoC systems, such as the Zynq7000 and Zynq Ultrascale+. Specifically, I will take a look into creating a basic AXI Lite IP to manage LEDs connected to the programmable logic (PL) side of the Zynq Ultrascale chip. Subsequently, I will construct a Vivado block design to incorporate the custom IP core and achieve communication between the PL and the processing system (PS) side of the Zynq Ultrascale.

Following this, I will export the hardware configuration and utilize Petalinux to generate a bootable image for an SD-Card. The focus here will be on demonstrating the development of a kernel driver tailored for the custom IP core's functionality under Linux. The IP Core will be designed to offer straightforward control over the LEDs, with user-accessible registers for toggling them on and off.

For this tutorial, I will be using the ALINX AXU2CGA Zynq Ultrascale+ board. However, alternative Zynq Ultrascale+ or Zynq7000 boards can be used, provided they feature LEDs connected to one of the PL banks.

Prerequisites:

Being familiar with Vivado, Petalinux, Verilog, AXI protocol and the C language, is very beneficial. Nevertheless I will try to briefly explain everything needed for this project.

The Workflow:

The toolset for this project includes Petalinux 2023.1 and Vivado 2023.1. This is the workflow that I will be following:

1. Create the custom IP-Core in vivado

2. Create the system block design

3. Synthesize, Implement, and Generate Bitstream

4. Export the hardware (.xsa)

5. Initiate a new Petalinux project

6. Import the.xsa into the Petalinux project

7. Write a kernel module for managing the custom IP-Core

8. Adapt the Petalinux project accordingly

9. Build the project

10. Generate the bootable image.

11. Develop a user application utilizing the kernel module to control the LEDs.

By following this structured approach, I am aiming to provide a simple comprehensive understanding of the development process from the hardware layer, where we work with registers and bits all the way to the user space in petalinux.

Part 1, Vivado and the custom IP-Core:

Let's begin by creating a new Vivado project without specifying any sources and selecting the board or SoC chip to be used. In my case, I'll go for the "xczu2cg-sfvc784-1-e" SoC and later employ a TCL script to define the board settings (I have no board definition file for the board I have so I am using a custom tcl script to set the different on board hardware components).

After the project is initialized, navigate to "create and package new IP" from the tools in the top bar, proceed to click next, then select "create a new AXI4 peripheral" and click next again. Assign a name; for instance, I'll choose "toggle_leds" and proceed to the next step. Now, opt for the interface type "Lite", set the interface mode as "Slave", configure the Data Width to 32 bits, and select 4 registers. Although we will not utilize all four registers, the minimum required is 4. Click next and proceed to "Edit IP", then finish. A new vivado project to customize and edit the IP will appear.

The AXI lite interface is ideal for managing and configuring registers in a system, therefor we are using it in our IP-Core. The slave interface of our IP-Core will be accessed by one of the general purpose master interfaces to interact with the PL side and initiate read and write transactions.

A new Vivado project window will appear, showcasing the AXI Lite Slave interface. I am using verilog here as I am more familiar with it (this can be done with VHDL as well). Under the design sources, we'll find the top level design along with the actual implementation of the AXI4 lite slave interface. (for data reading, writing, encoding, and decoding)

If you take a look at the implementation of the slave interface, you will find that we have four registers with the width 32bit that we can utilize for our own logic of this IP-Core.

So in my case I have a board that has 4 leds connected to one of the PL banks so I will be using one of these registers to drive the value for these 4 leds. Hence to control 4 leds we only need 4 bits and not the complete 32 bits.

The easiest way is to define a 4 bit output wire and connect it to one of these registers.

here we defined the output wire we need, it is defined as output because these 4 bits will be directly connected to the leds of the PL bank.

As we can see above we are assigning the leds output wire to one of the available slave registers we have(here I chose slv_reg0 and only the first 4 bits).

Now we need to modify the top level design, we are going to open the top level design and add the output wire we created and initiate it with the output wire we created in the slave interface.

Now after these changes move to the "Package IP" tab and choose the "Customization GUI" and click on "Merge changes from Customization GUI Wizard".

After merging and refreshing the new changes we can see how our IP-Core will look like. We can see the S00_AXI lite interface that we will use for the interaction with the PS side, we will basically use it to write to the register (reg_slv0) and remember that this values of these registers are also assigned to the output LEDS. Now we will basically choose "Review and Package" and choose to "Re-Package IP", our new custom ip core should now be avaliable in the IP cataloge.

Part 2, Vivado creating the block design:

Now in the main Vivado project we are going to "Create Block Design" and the add the Zynq UltraScale + MPSoC IP-Core or the Zynq7000 IP-Core (according to the board you have).

Then we add our custom IP "toggle_leds_custom_ip" and use the "Run Connection Automation" to connect both IP Cores together. The high performance master (HPM0) of the low power domain (LPD) should be connected to the slave interface of the toggle_led IP Core through the AXI Interconnect. The clocks and resets will be generated and connected automatically (in my case I am using 100MHz clock for the PL and an active low reset signal). Now we will right click on the output of the toggle_leds_custom_ip and basically make it external, because these will be connected to some of the pins of the PL bank of the SoC.

Once thats done, we will create an HDL Wrapper and then run synthesis followed by run implementation from the "Flow Navigator".

We will not generate the bitstream yet because the external connections are still not assigned to any pins. For that we can open the implemented design and observe that the LEDS_0[3:0] need to be assigned to some external pins. Now checkout the schematics of your own board and see to which pins are the Leds connected.

PS. the leds are connected in an active low configuration, this means the LEDs will switch on when the pins are set to low.

In my case Bank 24 pin AB13, AA12, Y12 and W13 have Leds connected to them. Now if we save our changes, it will ask to create a new XDC file. These changes will be saved as constraints. In future projects we will be adding/creating the constraints files directly from the sources. Now we can go ahead and generate the Bit Stream.

Once the generation of the bitstream is done, we can export the hardware description file "xsa" and include the bitstream in it in order to import it and use it under petalinux in the next step.

Part 3, Importing and using the hardware in Petalinux:

In this part we will use the AMD Xilinx petalinux SDK in order to create the Boot image that we will run on the hardware. We will import the xsa (including the bitstream) into a petalinux project and then develop the needed kernel driver as well as the user application to access our developed custom IP registers.

Please refer to my repository to install the AMD petalinux SDK on a linux system, in my case I will be using Ubuntu 22.04

Once the SDK is installed we can start by sourcing its settings by executing the following comand:

source /opt/petalinux/settings.sh

When installing petalinux using the script provided in my repo then the sourcing step is done automatically whenever a new bash terminal is opened.

Now we have the SDK activated and can create a new project, in my case I am using a zynqMP SoC so I will choose that platform while creating the project:

petalinux-create --type project --template zynqMP --name pl-leds

Now enter the created project:

cd pathToPetalinuxProject/pl-leds

And import the xsa file we exported from vivado (its usually exported to the root directory of the vivado project):

petalinux-config --get-hw-description /pathToXSA

This will import the xsa and call the menuconfig, for now we dont need to do any changes so we will just exit it.

In my case since I am not using any board support packages, and the zynqMP SoC I am using is CG series (only two cores) then I will need to delete the existing cpu interrupts and redefine them for the available cores only.

Under the following path of the petalinux project:

pathToPetalinuxProject/pl-leds/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi

And then add the following:

/include/ "system-conf.dtsi"
/ {

    pmu {
        /delete-property/ interrupt-affinity;
        interrupt-affinity = <&cpu0>, <&cpu1>;
    };

};

I do that because in the zynqmp.dtsi under pathToPetalinuxProject/pl-leds/components/plnx_workspace/device-tree/device-tree/zynqmp.dtsi the pmu node(platform management unit) is defined with interrupts for 4 cpu cores while the CG series has only two cpu cores, so the building of the project will fail with undefined refs. So in the system-user.dtsi I delete the interrupt nodes and redefine them for the two cores only.

Now we can build the complete petalinux project:

petalinux-build

After the project is built we can take a look at the compiled device tree.dtb, which is located under pathToPetalinuxProject/pl-leds/images/linux/system.dtb this is the compiled device tree for the entire system and it is a binary file so we can use dtc(device-tree-compiler) to decompile it and generate a readable device tree text.dts

dtc -I dtb -o dts -o system.dts system.dtb

If we take a look at the decompiled system.dts, we will find that the interrupt-affinity under the pmu node has only two values(in my case):

interrupt-affinity = <0x06 0x07>;

The 0x06 is the phandle of the cpu0 and the 0x07 is the phandle of cpu1.

Another interesting part to take a look at in the dts of a SoC device is the amba_pl node. The amba_pl node is where are our memory mapped ip cores are defined. Our toggle_leds custom ip core should be defined there.

amba_pl@0 {

#address-cells = <0x02>;
#size-cells = <0x02>;
compatible = "simple-bus";
ranges;
phandle = <0x72>;

toggle_leds@80000000 {
        clock-names = "s00_axi_aclk";
        clocks = <0x04 0x47>;
        compatible = "xlnx,toggle-leds-1.0";
        reg = <0x00 0x80000000 0x00 0x10000>;
        xlnx,s00-axi-addr-width = <0x04>;
        xlnx,s00-axi-data-width = <0x20>;
        phandle = <0x73>;
    };

};

In the example above we can see that the toggle_leds module is defined under the amba_pl node. We can see the properties of the toggle_leds module. It is defined at the physical address 0x80000000, (this also can be seen under the address editor in vivado). We can also see the clocks that this module is using, the register mappping and the compatible string. The auto generated compatible string "xlnx, toggle-leds-1.0" can be for example used to assign a kernel module to this hardware module (we will do that later to control our custom ip core).

Petalinux, PL-Leds kernel module:

In order to control hardware components under a linux system, we usually need a kernel module or a kernel driver. The custom ip core we created is considered as a hardware component.

We can start by creating a kernel module using a template that petalinux provides:

petalinux-create --type modules --name pl-led-driver --template c --enable

This will generate a kernel module template based on C and also enable it to be part of the kernel when compiling the complete petalinux project. It is found under the following path:

pathToPetalinuxProject/pl-leds/project-spec/meta-user/recipes-modules/pl-led-driver/files

Under files we can find the sources of the kernel module.

The auto-generated pl-led-driver.c already provied the basic functions to handle the probing and removing of the kernel module. The probing is responsible for mapping the physical address space of the module to a virtual address space. The remove will basically undo what the probing did.

There are two relevant kernel structures that we will use when writing a kernel module that are worth mentioning, the first one is the platform_driver and the second one is the file_operations.

The platform_driver is used to define platform device drivers and it has mandatory members like driver,probe and the remove functions and it can also include other optional ones for power management like shutdown, suspend and resume.

The file_operations is used to perform operations on the device, for example open, read, write, ioctl and many more operations. These operations are used to allow a user application to interface with the device.

PS: Remember that under the Linux OS we do not directly access the physical addresses but we have the MMU(memory management unit) that will handle mapping the physical addresses to virtual addresses we can then access. This is the opposite to baremetal/standalone programming where we can directly access the physical addresses.

The diagram above illustrates the process of how a user-level application interacts with a device driver through a system call, showing the flow between user-space and kernel-space. I mentioned kernel modules and device drivers in this context. Device drivers are specific kernel modules that control hardware devices.

Next we can do some changes to the template kernel module and implement our own system calls (ioctls) to write and read the registers of the toggle_leds module.

We will start to modify the template kernel module and assign it to the toggle_leds hardware device/ip core. To do that we need to use the of_device_id structure to match the device described in the device tree with the driver we created. Remember the compatiable string we saw in the device tree, we gonna use that in the kernel module in order to assign both to each other.

static struct of_device_id pl_led_driver_of_match[] = {
{ .compatible = "xlnx,toggle-leds-1.0", },
{ /* end of list */ },
}; MODULE_DEVICE_TABLE(of, pl_led_driver_of_match);

The compatible string in the kernel module is initialized to match the value of the compatible string of the device in the device tree, this way the kernel can bind the device with its driver.

Next we will use the unlocked_ioctl of the file_operations to define the switch on leds and switch off leds commands. We will also use the open from the file operations in order to be able to open the device and interact with it.

First we need to define the operations that the user can do when using the device driver. We have two very simple operations that are switching on and and switching off the PL Leds.

So we start to construct the ioctl by defining these two commands:

// --------------------- IOCTL commands ------------------------ //
#define IOCTL_DEVICE_DRIVER      'r' //ioctl unique identifier(magic number)
#define IOCTL_LEDS_ON            _IOWR (IOCTL_DEVICE_DRIVER, 0, int)
#define IOCTL_LEDS_OFF           _IOWR (IOCTL_DEVICE_DRIVER, 1, int)

Next we will define and implement the function that will utilize these commands. Here it will be very simple, we will have a switch for each command and each command. The base_addr is basically the starting address of the hardware device/ip core, simpley the offset we always need to use when writing or reading to the hardware device/ip core and from there we have four 32bit regs, where we only have utilized the first register that is reg0(see section vivado). Reg0 and its first four bits are connected to the PL leds. So here we basically set all four bits to 1 with the command IOCTL_LEDS_ON and we set them back to 0 with IOCTL_LEDS_OFF. We only trigger the last four bits, this means if the value of the physical address 0x80000000 is 0x00000000 it means the lights are on and if we set it to 0x0000000F then we have all four PL leds switched off.

static long leds_control_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
{
    struct pl_leds_local *lp = filep->private_data;
    unsigned int tempVal;

    switch (cmd)
    {
        case IOCTL_LEDS_ON:
	    iowrite32(0x0, lp->base_addr); // Active low configuration
	break;

	case IOCTL_LEDS_OFF:
	    iowrite32(0xf, lp->base_addr);
	break;
    }
    return 0;
}

We also need to remove the part related to the interrupt regestering since our IP-Core does not have any. We will also need to register the device as a charachter device, for that we need to allocate the minor and major numbers for the device as well as creating a class and then creating the device and regestering it under /proc/devices. Kernel module source code is found under my GitRepo. I will not compile the kernel module as built in to the kernel but as a.ko loadbale kernel module that can be loaded/unloaded anytime needed.(Its also possible to build it as a built in kernel module, same result is achieved in the end)

Now if I compile the kernel module and load it into the system then I will be sseing the following:

Next we can create a simple user level application that uses the kernel module and controls the pl leds.

The user level application will basically open a file handle to the device and then use the ioctls we defined. The user level application is found under my Git.

We can either build the user level application using petalinux or also compile it outside petalinx on the target or even on our working station and then copy it to the board.

I personally will use the an SD card to boot into my board, for that I will prepare an SD card and create two partitions. One partition as fat for booting and another ext4 partition for the rootfs. We can change this by changing the project config.

petalinux-config

The menuconfig will appear then change the Image packaging configuration from INITRAMFS to EXT4. next we can build the whole project again with the driver and the user level application (if you intend to have them in the generated image/fs)

petalinux-build

After the project is built and the kernel is compiled then we can generate the boot image.

 petalinux-package --boot --fsbl --fpga --pmufw --u-boot

Next we can copy the image.ub, boot.scr and the BOOT.BIN to the fat partition and the extract the rootfs.tar.gz to the ext4 partition.

Now we can insert the SD card into the board and set the boot pins to SD card if availabe. The system will boot and we can see the loading of the driver during the boot proccess(check the kernel buffer through dmesg) or load the kernel module.ko after the system is up. Next we can run the user level application, if it was built into the system then it will be found under /usr/bin or compile it externally and then copy and load it on the system. The user level application will simply switch off the leds and wait for a user entry then switch on the leds and wait again for user entry to switch the leds off again.

Congratulations! You just learned the process of creating a simple AXI lite ip core to control some leds on the PL side including the process of using petalinux to create a simple linux driver to set/unset the leds registers.