An FPGA Take on the Raspberry Pi: PetaLinux on the ZynqBerry

As I’ve mentioned in the past, I have a soft spot for well-marketed development boards (let’s be honest, I hoard dev boards in general) so when I came across an FPGA development board in a Raspberry Pi form factor with the name ‘ZynqbBerry’… I was sold immediately. I bought mine straight from the German company that designed them, Trenz Electronic.

If you’re not familiar with the various FPGA chipset families, the Zynq 7010 chip the ZynqBerry gets its name from is one of the SoCs (system-on-chip) from Xilinx’s lineup of FPGAs. Its defining feature is its dual-core Arm processor embedded into the fabric (logic gates) of the FPGA. This is advantageous in that now you have both a microcontroller and an FPGA in a single chip.

FPGA development gives you the design granularity necessary when you need to achieve properties such as low latency, parallel data processing, low power consumption, etc. However, because you’re designing at such a low level, you end up re-inventing the wheel in an especially painful way on basic functionalities such as serial communication interfaces. This is where having an Arm processor embedded into the FPGA is a huge advantage. Plenty of libraries, both in bare metal and embedded Linux, are out there to implement common interfaces such as SPI, I2C, UART, TCP/IP, etc. that are well written and tested (Side note: Soft-processors such as the MicroBlaze are an option that can be instantiated in an FPGA’s fabric, but we’ll just be focusing on the physical Arm processors in the Zynq in this article).

With my ZynqBerry in hand, I created a base Vivado project and a bare metal UART application that echoed back characters written to the UART on the micro-USB port as a test drive (posted here). After getting my feet wet, I quickly set my sights on expanding my design’s functionality to utilize the four regular USB ports and the Ethernet port. Xilinx’s SDK has bare metal drivers for both USB 2.0 and lwIP (light weight TCP/IP) that I had originally planned to use for my initial design. Although, when I started looking into the pinout of the Zynq 7010’s package on the ZynqBferry to figure out which package pins the ethernet and USB interfaces were routed to, I hit a firm roadblock. The four USB 2.0 and Ethernet ports are not connected directly to the Zynq chip. They are instead all routed to a SMSC LAN9514 chip, which is a USB 2.0 hub with an ethernet controller built in. To add a layer of complexity on top of that, the upstream USB PHY port of the LAN9514 is routed through a SMSC USB3320 hi-speed USB 2.0 ULPI transceiver in order to provide a high-speed parallel interface between the 5 ports and the Zynq (otherwise we’d have to implement a singular serial interface to continuously poll and handle the events on each of the ports — which would be significantly slower).

What this all boiled down to for me was that if I wanted to talk to the four USB 2.0 ports and the ethernet port from my bare metal application, I’d have to write my own ULPI interface driver in bare metal then figure out a way to convert that to talk to the lwIP driver and the USB 2.0 driver (remember what I was saying about re-inventing the wheel in an especially painful way?). Since bare metal had turned into a less than favorable option for this particular design, an embedded OS was the next option. Embedded Linux (implemented using PetaLinux for Xilinx chips) quickly worked its way to the top of my list given there are already drivers for the LAN9514 chip and ULPI-PHY interface.

A few side notes before jumping into my design flow in PetaLinux:

1 — After deciding on embedded Linux, I took note that the ZynqBerry only has 16MByte on onboard of on-board flash memory which meant that it’d need to boot from an embedded Linux image on an SD card (this is something that’s import to determine early on in your design as it will have an impact on how you configure your boot process).

2 — Xilinx provides board support packages for their reference boards within PetaLinux as a starting point for custom designs. What this really provides you is a starting point for your device tree for your Linux kernel. Personally, I found it easier to just start from scratch and edit the device tree myself (although with great power, comes great responsibility I learned…).

3— I’m running Vivado/SDK/PetaLinux version 2018.2 on Ubuntu 16.04 (Installation tutorial).

I personally like to keep all of my source files together (Vivado/SDK/PetaLinux) for a given project, so before I run the initial command to create the project (which creates the entire PetaLinux project file structure within whatever directory you’re in when you run the command) I change directories into the top level Vivado project folder. I then created my PetaLinux project targeting the Zynq and named it zynqberryOS.

Once the project had been created, I change directories into its top level folder to execute the configuration steps of building my embedded Linux image.

First things first, PetaLinux needs to know what hardware you have in your design. After generating the bitstream in Vivado, you export it to SDK where it is then compiled into a hardware description file (*.hdf). Based on the information this hardware description file contains, PetaLinux generates u-boot header files, the device tree source file, and enables the appropriate Linux kernel drivers.

This command will bring up the top system configuration menu in the form of an an ASCII GUI. The majority of these settings can be left as the default settings for the ZynqBerry, the only settings that need to be changed are the settings that configure the kernel and u-boot to point to the SD card.

The Subsystem AUTO Hardware Settings act as globals for system wide hardware settings, meaning that the tool is updating the device tree, u-boot configuration, and kernel configuration all in accordance to the information from these settings. There are two things that need to be changed here to configure the entire system to point to the SD card:

1 — Primary SD/SDIO needs to be set to ‘ps7_sd_1’ under SD/SDIO Settings

2 — Image Storage Media needs to be set to ‘primary sd’ under Advanced Bootable Images Storage Settings →dtb image settings

The only other tab that needs a couple of minor edits is the Image Packaging Configuration tab. The Root Filesystem Type needs to be set to ‘SD card’. Then if it’s not already, the device node of SD device should be set to ‘/dev/mmcblk0p2’. This tells the kernel to go look for the root file system in partition two of memory storage device zero. Since the SD card is the only memory storage device in this case, it will more than likely default to device zero when u-boot goes to look for it. We will verify this later…

The option for ‘Copy final images to tftpboot’ also needs to be unchecked since we are booting from an SD card and not over a network. You’ll see some random warning messages about tftpboot later on when the ZynqBerry is actually booting, but you can ignore them. It’s just because this option has been unselected here.

Exit the GUI and give it a little while to compile. If you need to come back and edit these settings at any point, you just need to run the ‘petalinux-config’ command from within the top level PetaLinux project folder. You only need to specify the hardware description file the first time after initial project creation.

Side note: the terminal window needs to be at least a minimum size in order for the ASCII GUI to be able to launch. So if you get a weird error message when you try to run ‘petalinux-config’, just make your terminal window large and rerun the command.

Most of the necessary device drivers have already been enabled based upon the previous settings in the top system configuration, but more specialized device drivers have to be enabled manually. Bring up the Kernel Configuration menu by running the following command:

This is where the drivers for the LAN9514 chip and ULPI-PHY interface are activated so that the four USB 2.0 ports and the ethernet port on the ZynqBerry can be utilized. Navigate to Device Drivers →Network Device Support →USB Network Adapters, and include the Multi-purpose USB Networking Framework option and the SMSC LAN95XX based USB2.0 10/100 ethernet devices option.

Again, exit the GUI and allow it time to compile. When it has completed, there is one more thing that needs to be done before building the entire project. Even though the drivers for the LAN9514 chip and ULPI-PHY interface have been enabled, it still needs to be added to the device tree blob. Within the PetaLinux project directory, /project-spec/meta-user/recipes-bsp/device-tree/files, is the device tree source include file (system-user.dtsi). This device tree file is the only user-editable one, the rest are auto-generated by PetaLinux and will either overwrite any edits made to them or cause your entire project to explode (yes, I did figure this out the hard way). Since we didn’t use a pre-packaged BSP, the system-user.dtsi file will be blank. Just add the code from the screenshot below, save, and close the file. I won’t go into detail here about exactly how this code works, but if you’re curious read over this link and this one.

Now that everything has been configured and compiled, its time to build the system image with the following command:

This step will take quite a while, especially the first time it’s run in a project. The status messages are very verbose by default so you’ll get plenty of feedback throughout the process.

The boot image for an embedded Linux image contains at the very least the u-boot binary files and first stage bootloader (FSBL). It is possible for the boot image file to contain everything including the FPGA bitstream, device tree, the Linux kernel image, and the root file system. Since the ZynqBerry’s Zynq chip is in a CLG225 package, SD Card boot directly from ROM bootloader is not supported. This means that the ZynqBerry’s on-board QSPI flash memory needs to be used for primary boot, and the SD card used for secondary boot. To achieve this, the boot image file for the ZynqBerry will just contain the FSBL, FPGA bitstream, and u-boot. The device tree, the Linux kernel image, and the root file system will be located on the SD card.

The packaging command to create the boot image file will end up looking something like this:

Note: the FSBL I’m using here is the same FSBL I used in my first bare metal application to test the UART. If you need a reference of how to create this FSBL, you can find it here. There is nothing special needed for this FSBL, it’s just the default Zynq FSBL application.

After the previous command has been run, the boot image file will be output into the /images/linux folder. Use the Program Flash Memory function in SDK with the special FSBL the ZynqBerry needs in order to have its on-board QSPI flash memory programmed over JTAG (the same link in the note in Step 5 details how to create this special FSBL) to program the BOOT.bin into the QSPI flash memory.

The SD card needs to be formatted with two different partitions. The first of which needs to be at least least 60MB in fat32, and it needs 4MB of unallocated space proceeding it. This is where the kernel image and device tree will live. The rest of the SD card needs to be formatted as ext4 and will be home to the root file system. Ext4 is a common Linux filesystem format, you can use ext3 but ext4 is the latest and greatest so I recommend sticking with it. Fat32 is one of the earliest filesystem types and boasts broad compatibility across many various operating systems. This is why it makes a good option for placing your kernel image and device tree on since the root file system you call could be pretty much anything. While your fat32 partition must be at least 60MB, it wouldn’t be a good idea to make it any larger than 4GB as it can only index a maximum of 4GB of files.

Each of the partitions on the SD card need a mounting point for files to be copied to them. Create one for the fat32 partition titled ‘BOOT’ and the other for the ext4 partition titled ‘rootfs’.

Mount the corresponding partitions of the SD card to each directory:

Copy the kernel image and device tree into the BOOT (fat32) partition and extract the root file system into the roots (ext4) partition:

Unmount each of the SD card partitions before removing the SD card from your computer:

You don’t have to launch Putty as a super user in order to run it, but if you want to create, save, or recall a saved configuration, you do need to run it as a super user.

The ZynqBerry creates a COM port during its boot process in order to provide an entry point into the u-boot environment and output status/error during the boot process. Knowing when to launch Putty in the boot process is the tricky part and took me a minute to figure out. You might also have to plug in the Zynqerry a few times to determine what COM port number it will settle on in your system to save into a Putty configuration.

Note: Since this is a Zynq chip we are working with, the default baud rate is 115200.

Install the SD card into the ZynqBerry and plug it in to your computer via its JTAG port (on the micro-USB connector). You’ll see the red status LED near the micro-USB blink rapidly at first, then switch to a slower pace before stopping. Wait until the red LED is blinking at the slower pace to launch the Putty session. The COM port has not been created yet when the LED is blinking rapidly, but once the LED has shut off there is nothing being output on the COM port to see any longer. The Putty session also needs to be launched early in the time frame in which the LED is blinking slowly, as this is the time frame in which u-boot allows itself to be halted for editing. When you launch Putty at the right time, you’ll see a countdown happening, this is the countdown before u-boot will continue on with the boot process. Press any key during this countdown to halt the boot process and enter the u-boot environment.

There are a few things in the ZynqBerry’s u-boot environment that need to be changed. A couple of boot arguments and boot commands need to be added to tell the kernel that the partition the root file system is on is ext4 format, and where to load the kernel image and device tree from.

First things first, we need to check to see that the SD card actually showed up as device 0 as we predicted back in Step 2 when we set the device node of the SD card to /dev/mmcblk0p2 in the top system configuration menu.

Run mmc list in the command prompt. If the device shows up under node 1, you’ll need to start back at Step 2 and change set the device node of the SD card to /dev/mmcblk1p2.

To tell u-boot where to load the kernel image and device tree from, run the following commands:

setenv cp_dtb2ram ‘fatload mmc 0 ${dtbnetstart} ${dtb_img}’ setenv cp_kernel2ram ‘fatload mmc 0 ${netstart} ${kernel_img}’ setenv default_bootcmd ‘run cp_kernel2ram && cp_dtb2ram && bootm ${netstart} - ${dtbnetstart}’

Something I found to be interesting was that I found the kernel whining about the root file system partition on the SD card being in ext4 instead of ext3, despite the fact that ext4 is the current standard Linux filesystem type and Xilinx’s user guide for PetaLinux (UG1144) tells you to format that partition of the SD card as ext4. I got rid of this peskyness by modifying my boot arguments as such:

setenv bootargs ‘console=ttyPS0,115200 earlyprintk root=/dev/mmcblk0p2 rootfstype=ext4 ru rootwait’

Side note: if your SD card shows up as device 1 instead of device zero, everywhere you see ‘mmc 0’ or ‘mmcblk0’ in my commands above you’ll need to change to ‘mmc 1’ and ‘mmcblk1’.

Once you have made all of these changes, save the ZynqBerry’s boot environment by writing it to its flash memory with the command ‘saveenv’. You can print out all of the current u-boot settings to check them at any time with the command ‘printenv’.

After saving the boot environment to the ZynqBerry’s flash memory, it’s ready for its maiden boot voyage! Simply run the command ‘boot’ in the u-boot editor and watch the boot process step through its paces. If you run into a problem don’t only look at the last message printed out on the COM port, be sure to scroll back and read all of the messages from the beginning of the boot process all the way up to wherever it got hung up. I had accidentally told u-boot to boot from the second Arm processor when I meant to tell it to boot from the first Arm processor on my initial attempt at this. It got hung up in a weird place in the boot process and the last message printed out had me chasing my tail thinking my SD card was corrupt because the message was saying it couldn’t read an ext4 format. Once I started scrolling back through the messages on the COM after trying my third SD card and getting the same result, I finally found a few messages that told me what was really going on towards the beginning of the boot process.

If you didn’t set your own custom login when you configured the kernel back in Step 3, the default login is username = root and password = root. Definitely go back and reconfigure your kernel to set your own custom login if you plan to use your ZynqBerry on a network.

As you can see, once I got logged in, I used the list command to verify my file system structure was all there. It’s also just a satisfying sanity check to see everything is where you think it should be. I know this is quite a long read, but I really wanted to demonstrate some of the thought process behind embedded system design and how obstacles influence the design choices made to achieve a solution. The roadblock I hit when I initially attempted to write a bare metal driver to control the four USB ports and ethernet port that lead me to choose to create an embedded Linux image is a perfect example of the kind of design choice changes embedded system designers have to make on a regular basis.