Live Camera Feed Webpage on the Ultra96

The Ultra96v2 is one of the best FPGA development boards for wireless applications and IoT projects. As I've been working through new project ideas, camera applications of some sort kept coming to mind. So I decided to start with a simple webpage application with a live camera feed.

Vivado/PetaLinux 2019.2 BSP for Ultra96V2

I've covered how to create BSPs for Avnet boards in a past project on creating a Custom Webserver on the MiniZed so you can follow that and just run one of the make Ultra96v2 scripts instead, or I have also covered how to build a new PetaLinux project from the Ultra96v2 oob BSP as well. I attempted to make this project my first version 2020.1 project, but PetaLinux 2020.1 simply wouldn't work right for me so I'll have to save that for another day. This project will be all version 2019.2 for Vivado, Vitis, XRT, and PetaLinux.

Create a new PetaLinux project using the BSP just created for the Ultra96v2.

petalinux-create -t project -s /<path to BSP>/ultra96v2_oob_2019_2.bsp

Change directories into the project and build it.

cd ./ultra96v2_oob_2019_2
petalinux-build

If you get a build failure due to the arduino-toolchain package, see my fix in my past project here.

Package the boot binary image (BOOT.BIN) for the Ultra96v2.

petalinux-package --boot --fsbl ./images/linux/zynqmp_fsbl.elf --fpga ./images/linux/system.bit --pmufw ./images/linux/pmufw.elf --u-boot

You can alternatively use the boot binary, kernel, and root filesystem in the 'pre-built' directory of the PetaLinux project. However this method would be required if you want to add any kernel drivers or root filesystem packages to the embedded Linux image.

Prep the SD Card

Using an SD card at least 4GB in size, partition a 500MB with 4MB of free space preceding it as FAT32. Format the remaining space on the SD card as EXT4. I personally prefer to use the Gparted GUI in Ubuntu to do this, but it can easily be done from the command line as well.

If you haven't previously, create mounting point folders for the two partitions of the SD card. One for the boot partition (FAT32) and a second for the root filesystem partition (EXT4):

sudo mkdir /media/BOOT/
sudo mkdir /media/rootfs/

Mount the SD card partitions to the appropriate mounting point folder:

sudo mount /dev/sdc1 /media/BOOT/
sudo mount /dev/sdc2 /media/rootfs/

Extract the packaged root filesystem to the EXT4 partition of the SD card and run the 'sync' command immediately afterwards to ensure all of the data has been written to the SD card.

sudo tar -xf /<petalinux project dir>/ultra96v2_oob_2019_2/images/linux/rootfs.tar.gz -C /media/rootfs/
sync

Copy the kernel image, device tree, and boot image to the FAT32 partition and again run the 'sync' command immediately afterwards.

sudo cp /<petalinux project dir>/ultra96v2_oob_2019_2/images/linux/BOOT.BIN /media/BOOT/
sudo cp /<petalinux project dir>/ultra96v2_oob_2019_2/images/linux/image.ub /media/BOOT/
sudo cp /<petalinux project dir>/ultra96v2_oob_2019_2/images/linux/system.dtb /media/BOOT/
sync

Finally, unmount the partitions of the SD card before unplugging it from your PC:

sudo umount /media/BOOT/
sudo umount /media/rootfs/

Boot Up the Ultra96v2

Install the SD card into the Ultra96v2 and verify the boot switch (SW3) is set to SD boot (switch 1 = OFF, switch 2 = ON), then press the power button (SW4) to power up the board.

For the UART serial console of the Ultra96v2, I'm using the USB to JTAG/UART Pod Adapter Board connected to the the UART and JTAG headers (J1 & J3).

To connect the Ultra96v2 to a wireless network, the 2019.2 BSP requires you to use the Wi-Fi setup webpage to select a network and enter the credentials to log in (follow the instructions in section 13 on page 24 of the Getting Started Guide for the Ultra96V2 if you're not familiar with this process). Reboot the Ultra96v2 after getting the success message that the network was added, do not use the "Check Connection" button. This is just a glitch I've found with the 2019.2 BSP.

root@ultra96v2-oob-2019-2:~# reboot

I have noticed in the 2019.2 build for the Ultra96, there is a slight delay in the board actually getting connected to the network after boot up and when it initially connects, the network configuration does show the locally assigned IPV4 address either. This is my only complaint about the Ultra96, I find myself constantly fighting to establish a stable Wi-Fi connection. From what I've found, the board will connect to the last network the Wi-Fi setup webpage connected to if that network is available.

After rebooting the Ultra96v2, wait for a couple of minutes then test the network connection by pinging either your PC or another website. I typically try pinging google.com to verify external internet connection.

root@ultra96v2-oob-2019-2:~# ping google.com

Plug in webcam & init drivers

Plug in the web camera to one of the USB type A ports (J8 & J9) and allow a few moments for the Ultra96v2 embedded Linux image to detect it. You'll see some console output from the UART as this is happening. After this finishes printing, us the usb-devices command to display all USB devices on the system.

root@ultra96v2-oob-2019-2:~# usb-devices

A list of all the USB buses will appear, including the actual USB controllers and the devices connected to them. By looking at the listed product, vendor, and drivers you can see if the system has successfully detected the web camera.

Now that the system has successfully booted up and detected the web camera, we can move on to writing the Python scripts to utilize it and the custom web page to display the images from it.

The Web Application

The OpenCV-Python library (cv2) is only available in the Python3, and the custom project template in the Ultra96v2 webpage runs scripts with Python 2.7, so I created the script using the 'Create Project' option on the Custom Content page and tested the base web camera Python script from the UART serial console of the Ultra96v2 to be able to execute it using Python3.

root@ultra96v2-oob-2019-2:/usr/share/ultra96-startup-pages/webapp/templates/CustomContent/custom# python3 ./webcamcapture.py

This script the web application is executing to control the web camera is simply capturing one image per every POST or GET request made to the webpage. Which is once every ten seconds as will been seen in the front end HTML of the web page.

Web Camera Python Script:

import os
import cv2
import time

camera = cv2.VideoCapture(0)
time.sleep(0.1)
    
(success,reference) = camera.read()

cv2.imwrite('/usr/share/ultra96-startup-pages/webapp/static/images/single_frame.jpg',reference)

camera.release()
cv2.destroyAllWindows()

The script starts by creating a VideoCapture object that is passed the index number of the video device, which is zero in this case since there is only one video device plugged into the system. After a tenth of a second wait, the read() function triggers the camera to capture an image and reads it in. This returns a tuple (return value, image) which the return value indicates whether the capture was success or not and the captured image is the second parameter returned.

The image is written to the specified filename single_frame.jpg in the /usr/share/ultra96-startup-pages/webapp/static/images/ directory using the cv2.imwrite() function.

Then when the processing is done, release the capture and, with the cv2.destoryAllWindows(), close the window created for the image captured.

When the script is running, the green status light on my web camera lit up to indicate the Ultra96v2 was successfully communicating with it to capture images.

For the web application front end, I simply have it displaying the image captured from the web camera with the page refreshing every 10 seconds to display an updated image.

Navigate to the the Custom Content tab of the Ultra96 web server, and select the "Edit Webapp" option. To create the front end for the custom web page, click the "Create Front End" button and the HTML text editor will appear.

Front end code:

{% extends "Default/default.html" %}
{% block content %}

<div class="page-header">
  <h1 class="display-4"><b>{% block title %}USB Web Cam Feed{% endblock %}</b></h1>
</div>

<!-- Start adding your code below here -->

<meta http-equiv="refresh" content="10">
<img src="{{ url_for('static', filename='images/single_frame.jpg') }}"/>

<!-- Stop adding your code here -->

{% endblock %}

After adding the front end HTML, click the "Save File" button and specify a name for the file. After saving the file, click the "Back" button to return to the previous screen.

To create the back end for the custom web page, click the "Create Back End" button and the text editor will appear. The back end is written in Python and the base code will already be filled in. The only thing that needs to be added is the call to run the project script to capture a new image on every request.

Back end code:

@app.route("/webcam.html", methods=["GET", "POST"])
def webcam():
    if request.method == "POST":
        proc = subprocess.Popen('python3 /usr/share/ultra96-startup-pages/webapp/templates/CustomContent/custom/webcamcapture.py',stderr=subprocess.STDOUT,shell=True)
        err = proc.communicate()
        
        return render_template("CustomContent/custom_front_end/webcam.html")
    else:
        proc = subprocess.Popen('python3 /usr/share/ultra96-startup-pages/webapp/templates/CustomContent/custom/webcamcapture.py',stderr=subprocess.STDOUT,shell=True)
        err = proc.communicate()
        
        return render_template("CustomContent/custom_front_end/webcam.html")

Again, click the "Save File" button and specify the same name for the backend file as was given to the front end file. After saving the file, click the "Back" button to return to the previous screen.

Check the "Include" box then click "Reload Webapp" to add the new web page to the wevserver on the Ultra96v2.

After reloading the webpage to include the new custom web page, I found the 2019.2 BSP also contains the same glitch that the main webserver.py script calls the main function prior to the app.route definition of the new web page. So I still have to use a text editor to comment out the first call to main in webserver.py every time after reloading the web app.

root@ultra96v2-oob-2019-2:~# vi /usr/share/ultra96-startup-pages/webapp/webserver.py

As you can see, the first call to main on lines 919 & 920 need to be commented out.

After saving and exiting the text editor, manually reboot the Ultra96v2:

root@ultra96v2-oob-2019-2:~# shutdown -h now

Then use the power button (SW4) to power the board back on.

Navigating back to the Custom Content page of the Ultra96 web server, you'll see a 'view' button next to the newly added web page.

After clicking the view button, you'll see the feed appear and the page refreshing on its own every 10 seconds with an updated image. This solution is a very simple, low-bandwidth video feed that works great for simple monitoring system projects that is easily expanded on and can be adapted to a variety of different projects.