Design Flow for a Custom FPGA Board in Vivado and PetaLinux

A little while back, a Raspberry Pi form factor FPGA board called the ZynqBerry caught my eye and I spent some time with it to bring it up as a ready-to-go tool in my arsenal of development boards. I quickly found that I needed an embedded Linux image in order to utilize the Ethernet port and four USB ports on it which led to previous post here. I’ve been happy with it since, but with all the hype from the new Raspberry Pi 4 I was reminded that I needed to pick it back up and finish the device tree build out for the rest of the peripherals such as the HDMI port, etc. This prompted me to want to put together an outline of my design flow for FPGA development from start to finish all in one place. Now while this specific design flow is Xilinx-based, I’ve found that the main ideas can be applied to other chip sets in other IDEs to help adjust to the new environment faster (for example — see my first crack at using Lattice Semiconductor’s FPGA and IDE for the first time).

In Vivado, there are a ton of pre-packed IP (intellectual property) blocks to cover a ton of basic functionalities for you to utilize such that you can focus more so on the custom parts of your design instead of re-inventing the wheel over and over again on things like UART drivers, SPI interfaces, etc. The block design is also where you will instantiate your soft or hard processors such as the MicroBlaze (Xilinx’s proprietary soft processor) or ARM processors (if you’re using one of Xilinx’s Zynq chips). Now this step is optional if your design is purely your own custom HDL. Xilinx’s built-in IP repository is also available at the HDL level, but I really recommend using it in the block design if you are going to use it since the block design editor in Vivado offers automated design assistance in wiring blocks together as you drop them in.

To create a block design in a project, just select the option ‘Create Block Design’ under the IP Integrator tab in the Project Manager menu. After that, it’s drag and drop!

Another cool thing about the block design in Vivado is that you can package an entire project into its own IP block and place it into a local repository to use in other designs. After looking into Trenz Electronic’s support page for the ZynqBerry, I found that they had created quite a bit of their own custom IP for the ZynqBerry. After downloading the IP library from Trenz’s website, I added it to my project in Project Manager under ‘Settings’ → ‘IP’ → ‘Repository’ → ‘Add Repo’. This now let me run the TCL script from Trenz to recreate the ZynqBerry’s block design containing these custom IP blocks.

In order to recreate a block design from a TCL script like this, don’t manually create a new block design. Instead, straight from the TCL console in an empty project run the following command:

source /yourBlockDesign.tcl

Once the block design is complete, whether it’s created by hand or recreated from a TCL script, I run validation on the design by clicking the ‘verify’ button in the top menu bar. Once the design passed validation, I save and close the block design.

In order for the block design and HDL to interface, a top level HDL wrapper is needed. This top level wrapper will instantiate an instance of the block design that then makes it available to any other instances of HDL modules. It’s important to remember the only signals that will be directly available in the wrapper from the block design are those signals that were made external by bringing them out to a pin in the block design.

To create a top level wrapper, right click on the block design in the Sources tab and select the ‘Create HDL Wrapper…’ option.

There are two options when creating a new HDL wrapper: allow Vivado to manage and auto-update it, or manually configure it as desired. This option is relevant to if/when the block design needs to be updated later on. The most error-proof method I have personally found to go about this is that I select the option to allow for Vivado to manage the HDL wrapper, then create my own module in Project Manager by selecting ‘Add Sources’ → ‘Add or Create Design Source’ and I simply copy+paste the instantiation from the auto-generated wrapper file into my own. I then set my own file as the new top level file for the design and disable (NOT delete) the wrapper auto-created by Vivado. If I need to update the block design, I re-enable the auto-created HDL wrapper and set it back as the top before going into the block design to modify it. This super convoluted, indirect way to do this is hyper specific to the Vivado IDE. There is so much happening in the background with every action in the GUI, I’ve learned it’s best to minimize interruptions to those processes as much as possible and this is the best way I’ve found to accomplish that for the block design.

Once the top level HDL wrapper is in place, I go about writing my Verilog/VHDL as normal and instantiating it as necessary in my design. For this project on the ZynqBerry, I just left the auto-created wrapper as the top level module since I didn’t write any custom HDL for it. All of the design is contained with in the block design.

There are a couple of different ways that I go about this depending on whether I’m creating a new constraints file or importing an existing constraints file. If I’m creating new constraints, I run synthesis and use the IO Planning GUI in the synthesized design to route my signals to pins. If I’m importing existing constraints like I am for the ZynqBerry, I go to Project Manager, select ‘Add Sources’ → ‘Add or Create Constraints’ then import the existing file. In Vivado, it treats anything imported into that ‘Constraints’ folder as a whole set. This means you don’t have to cram everything into one file, as you constraints can get pretty lengthy for some designs. I personally like to split it up by having one file for my timing constraints/clock creations, then I put all of my pinouts in a second file. When you do have multiple files in your constraints set, you do need to specify one as your ‘target’ constraints file (if there is only one, then Vivado sets it as the target by default). I set my timing/clock creation constraints file as my target (right click on file → ‘Set as Target Constraints File’).

When I downloaded the constraints set to the ZynqBerry from Trenz, I noticed that they had done something similar and they had also made a separate file for each peripheral. I might start doing the same in the future for bigger designs since it made everything so readable.

Once the constraints have been set, synthesis and implementation needs to be ran to build the design and route it through the targeted chip. This step is a lot of watching and waiting. If there are any errors or critical warnings, the resulting output error code and Google are your best friend.

After the design is routed in implementation, the setup and hold timing of all paths within the FPGA fabric are calculated. If the design does not meet proper timing constraints, a critical warning is generated after implementation is complete. There are a multitude of techniques to fix timing issues depending on their root cause. In Vivado specifically, you can find a ton of resources for how to fix timing issues in DocNav if you search ‘Timing Closure & Design Analysis’.

Once any errors and critical warnings are resolved, the design is ready to be packaged up into a bitstream to export to SDK. If you run into any errors here, it’s typically due to improper voltage level specifiers or some other mismatch in your pinout constraints. Don’t ignore any critical warnings, especially here, because it will come back to bite you in expected behavior later on.

Now that the hardware design is complete and verified, the next step is to export it to SDK where the appropriate embedded software can be added to the design. To do this go to File → Export Hardware → check the box to include the bitstream and leave the location as . Technically, you can export your hardware definition and create the SDK workspace wherever you want to, but as I mentioned previously, the most error-proof way is to allow Vivado to place things where it wants to. To launch SDK after exporting the hardware design, go to File → Launch SDK → and again, leave location options as .

Even though I am ultimately using PetaLinux to create an embedded Linux image for the ZynqBerry, the first stage bootloader to launch the Linux kernel is a bare metal application that is created in SDK. The ZynqBerry actually needs two first stage bootloaders: the normal FSBL to store in flash memory and launch the Linux kernel located on the SD card, and a second special ZSBL to bring up the ZynqBerry via JTAG to allow for us to program the normal FSBL into flash memory. This JTAG FSBL is also used by the system debugger in SDK if debugging a bare metal application (which is done via JTAG). I’ve covered the details for creating these two FSBLs for the ZynqBerry in the past here.

When creating a new PetaLinux project, I personally like to create it in the same folder as my Vivado project, so I will change directories into that folder prior to running the following command to create the project.

petalinux-create --type project --template zynq --name 

PetaLinux will create a top level folder the same name as the project name you pass it with the ‘-name’ option to place the project in. Change directories into this project folder before running any of the project configuration commands.

cd ./

The hardware settings is where the kernel is configured to boot from the SD card for this design. The following PetaLinux command imports the hardware description of the design from SDK and launches a GUI to configure the hardware settings. All of these settings are the same between this full build out of the ZynqBerry and my previous post where I was just focused on interfacing with the Ethernet port, which you can find here under step 2.

petalinux-config --get-hw-description 

Once the hardware settings are configured (these settings are just making PetaLinux aware of the hardware you’ve already chosen, so make sure your settings make sense with what you’ve designed in Vivado) the next thing to do configure is the kernel. The Linux kernel is responsible for starting up and managing the resources processes and peripherals applications in the OS are using. In a sense, the kernel is basically the glue between the operating system and the hardware.

petalinux-config -c kernel

The kernel now needs to be told what hardware is available to it such as the USB and Ethernet PHYs, sound driver (ALSA), graphics for the HDMI port, and the I2C drivers for each. Again, the above PetaLinux configuration command will launch a GUI to make these selections in. The ZynqBerry requires the following configuration:

Finally, the root file system for the ZynqBerry just needs a few things enabled specifically around being about to use the Advanced Linux Sound Architecture (ALSA) and the I2C package library.

petalinux-config -c rootfs

With everything configured in the kernel and root file system settings, the next thing to do is build out the device tree. The device tree in Linux serves as a database of properties of all the hardware devices for the kernel to use while it manages how/when the OS accesses each device. The structure of the device tree file follows a simple node with given properties format.

/include/ "system-conf.dtsi"
/ {
};

/ {
    #address-cells = ;
    #size-cells = ;

    reserved-memory {
        #address-cells = ;
        #size-cells = ;
        ranges;
        hdmi_fb_reserved_region@1FC00000 {
            compatible = "removed-dma-pool";
            no-map;
            // 512M (M modules)
            reg = ;
            // 128M (R modules)
            //reg = ;
        };

        camera_fb_reserved_region@1F800000 {
            compatible = "removed-dma-pool";
            no-map;
            // 512M (M modules)
            reg = ;
            // 128M (R modules)
            //reg = ;
        };
    }; 

    hdmi_fb: framebuffer@0x1FC00000 {           // HDMI out
        compatible = "simple-framebuffer";
        // 512M (M modules)
        reg = ;    // 720p
        // 128M (R modules)
        //reg = ;   // 720p
        width = ;                         // 720p
        height = ;                         // 720p
        stride = ;                  // 720p
        format = "a8b8g8r8";
        status = "okay";
    };

    camera_fb: framebuffer@0x1F800000 {         // CAMERA in
        compatible = "simple-framebuffer";
        // 512M (M modules)
        reg = ;    // 720p
        // 128M (R modules) 
        //reg = ;   // 720p
        width = ;                         // 720p
        height = ;                         // 720p
        stride = ;                  // 720p
        format = "a8b8g8r8"; 
    };

    vcc_3V3: fixedregulator@0 { 
        compatible = "regulator-fixed";
        regulator-name = "vccaux-supply";
        regulator-min-microvolt = ;
        regulator-max-microvolt = ;
        regulator-always-on;
    };
};

&qspi {
    #address-cells = ;
    #size-cells = ;
    status = "okay";
    flash0: flash@0 {
        compatible = "jedec,spi-nor";
        reg = ;
        #address-cells = ;
        #size-cells = ;
        spi-max-frequency = ;
        partition@0x00000000 {
            label = "boot";
            reg = ;
        };
        partition@0x00500000 {
            label = "bootenv";
            reg = ;
        };
        partition@0x00520000 {
            label = "kernel";
            reg = ;
        };
        partition@0x00fa0000 {
            label = "spare";
            reg = ;
        };
    };
};

/*
* We need to disable Linux VDMA driver as VDMA
* already configured in FSBL
*/
&video_in_axi_vdma_0 {
    status = "disabled";
};

&video_out_axi_vdma_0 {
    status = "disabled";
};

&video_out_v_tc_0 {
    //xilinx-vtc: probe of 43c20000.v_tc failed with error -2
    status = "disabled";
};

&gpio0 {
    interrupt-controller;
    #interrupt-cells = ;
};

&i2c1 {
    #address-cells = ;
    #size-cells = ;

    i2cmux0: i2cmux@70  {
        compatible = "nxp,pca9544";
        #address-cells = ;
        #size-cells = ;
        reg = ;

        i2c1@0 {
            #address-cells = ;
            #size-cells = ;
            reg = ;

            id_eeprom@50 {
                compatible = "atmel,24c32";
                reg = ;
            };
        };

        i2c1@1 {    // Display Interface Connector
            #address-cells = ;
            #size-cells = ;
            reg = ;
        };

        i2c1@2 {    // HDMI Interface Connector
            #address-cells = ;
            #size-cells = ;
            reg = ;
        };

        i2c1@3 {    // Camera Interface Connector
            #address-cells = ;
            #size-cells = ;
            reg = ;
        };
    };
}; 

/{
    usb_phy0: usb_phy@0 {
        compatible = "ulpi-phy";
        #phy-cells = ;
        reg = ;
        view-port = ;
        drv-vbus;
    };
};

&usb0 {
    usb-phy = ;
};

/*
* Sound configuration
*/

/{
    // Custom driver based on spdif-transmitter
    te_audio: dummy_codec_te {
        compatible = "te,te-audio";
        #sound-dai-cells = ;
    };

    // Simple Audio Card from AXI_I2S and custom XADC audio input 
    // and PWM audio output cores
    sound {
        compatible = "simple-audio-card";
        simple-audio-card,name = "TE0726-PWM-Audio";
        simple-audio-card,format = "i2s";
        simple-audio-card,widgets =
            "Microphone", "In Jack",
            "Line", "Line In Jack",
            "Line", "Line Out Jack",
            "Headphone", "Out Jack";

        simple-audio-card,routing =
            "Out Jack", "te-out",
            "te-in", "In Jack";

        simple-audio-card,cpu {
            sound-dai = ;
        }; 

        simple-audio-card,codec {
            sound-dai = ;
        };
    };
};  

&audio_axi_i2s_adi_0 {
    compatible = "adi,axi-i2s-1.00.a";
    reg = ;
    clocks = , ; // FCLK_CLK0, FCLK_CLK3
    clock-names = "axi", "ref";
    dmas = ;
    dma-names = "tx", "rx";
    #sound-dai-cells = ;
};

/*
* We need to disable Linux XADC driver to use XADC for audio
* recording
*/
&adc {
    status = "disabled";
};

By looking at this device tree and corresponding comments you can start to get an idea of how a device tree works. Each peripheral on the board has it’s own node defining its register addresses and in some cases, which driver to link it to in the root file system.

With everything configured and the device tree built, the project needs to be built which will call all of the various compilers needed to create the final output files for the kernel image and root file system.

petalinux-build

Since the ZynqBerry will be booting from an SD card, the only things that need to be packaged into the boot image file are the normal first stage bootloader (NOT the one for JTAG bring up), FPGA bitstream, and u-boot. If the board were being booted from some sort of on-board memory, then this boot image file would also include the kernel, device tree, and root file system.

petalinux-package --boot --fsbl  --fpga  --u-boot

The boot image file will live in the on-board flash memory of the ZynqBerry, which can be loaded using SDK using Program Flash Memory. This is where the special JTAG FSBL comes in to bring up the ZynqBerry initially to be able to program the QSPI flash memory.

The packaging command from the previous step will output to the BOOT.bin to /images/linux.

Step 14 — Prep and load SD card with Linux kernel and root file system

Now that the boot image file is loaded into the FPGA’s flash memory to point to the rest of the boot process to the SD card, the SD card needs to be prepped accordingly. The files that will live on the SD card require two different file system formats. The device tree and kernel image work best in a fat32 format since fat32 is so widely compatible with various operating systems. The root file system however needs the better performance and reliability from a format such as the latest and greatest extended filesystem, ext4.

I personally like to use GParted to format my SD cards, and I also like to leave a few MB of unallocated space ahead of the first partition. This came about on the many Raspberry Pis and countless Pi images I’ve flashed, when I was having issues with the image going corrupt quickly and needing to reformat/reflash the SD card on a regular basis. After much Googling, and trail and error, I found that leaving anywhere from 2MB — 8MB of unallocated space ahead of the first partition on an SD card led to much better stability. I found sweet, sweet vindication on this when I found in UG1144 (Xilinx’s official documentation for PetaLiunx) that Xilinx recommends to leave 4MB of unallocated space ahead of the first partition on an SD card.

The first partition is the fat32 where the kernel image and the device tree it will reference will live (UG1144 recommends a minimum of 60MB), and the second partition is the ext4 for the root file system. At least 4GB is recommended for the base root file system PetaLinux will generate, but it’s important to keep in mind what the end use is going to be here. Depending on the number and type of custom applications to be developed, the overall memory needed will increase.

Once the SD card has been formatted, create a directory to mount it to and mount it.

sudo mkdir /media/BOOT
sudo mkdir /media/rootfs

sudo mount /dev/ /media/BOOT
sudo mount /dev/ /media/rootfs

Copy over the kernel image and device tree to the BOOT directory:

sudo cp /images/linux/image.ub /media/BOOT
sudo cp /images/linux/system.dtb /media/BOOT

Exact the root file system into the rootfs directory:

sudo tar xvf /images/linux/rootfs.tar.gz -C /media/rootfs

And finally unmount the SD card:

sudo umount /media/BOOT
sudo umount /media/rootfs

The Universal Boot Loader, u-boot, is the primary bootloader for Linux that the Zynq FSBL (the bare metal application created in SDK) kicks off in the overall boot sequence. For this design in particular, since the ZynqBerry is being booted from the SD card, u-boot needs to have the block number of the SD card configured into it for where to load the device tree and kernel from. We’ll then also tell u-boot to first load the kernel, followed by the device tree for the kernel to index the hardware, then continue on with the boot up.

To enter the u-boot editor environment, you have to catch it at just the right moment in the boot sequence, as you can only enter the u-boot editor during the time in which u-boot is the active component in the boot sequence. On the ZynqBerry I found that this was after the red status LED transitioned from blinking rapidly upon initial power up to blinking slower, but even then there is still only a limited window before the kernel takes over and completes the boot sequence. This took a lot of trial and error for me to figure out, and it was by far where I was stuck the longest the first time I was doing this.

The u-boot editor environment is accessed via a COM port, so it is the first thing that creates a COM port in the boot sequence. During my trial and error on the ZynqBerry, I was able to correlate the u-boot starting with the blink rate of the red status LED based on how early I was able to get Putty to connect to the ZynqBerry’s COM port after applying power.

Once in the u-boot editor, the setenv command is used to modify the necessary settings. I then echo back all the current settings with printenv to verify everything before saving with saveenv.

setenv bootargs ‘console=ttyPS0,115200 earlyprintk root=/dev/mmcblk0p2 rootfstype=ext4 ru rootwait’

setenv cp_dtb2ram ‘fatload mmc 0 ${dtbnetstart} ${dtb_img}’
setenv cp_kernel2ram ‘fatload mmc 0 ${netstart} ${kernel_img}’
setenv default_bootcmd ‘run cp_kernel2ram && cp_dtb2ram && bootm ${netstart} - ${dtbnetstart}’

printenv

saveenv

Before editing these setting however, the block number of the device (SD card) needs to be verified. The ‘mmclist’ command will list all al the available devices u-boot sees. The set environment commands above all assume a device node of 0, but if the SD card’s node number is different then the u-boot boot arguments and boot command will need to be changed to reflect that, as well as the kernel will also need to be reconfigured to look for the proper device node number.

With everything configured and ready to go, the boot sequence can be commanded to continue from the u-boot editor environment:

boot

If all goes well, a login prompt is presented. By default, the username and password in PetaLinux images are both ‘root.’ And this ZynqBerry is ready to go!

Like always, I have created a GitHub repo with the specific design files for my project you can find here!