Helping Out My Local Coffee Shop During COVID Using FPGAs

Objective

In my last project post, I detailed how I set out to help my local coffee shop to adapt to the new way of digital, contactless menus during the pandemic. A custom Flask web server with a webpage displaying the coffee shop's menu, set on a port to allow access external of the coffee shop's network with a link embedded in a QR code is the bulk of the solution for them, but there was a little more that they needed.

Like many coffee shops, a single batch of pastries are made and delivered to the shop for the day. This means that as the day progresses, certain menu items are not available as they sell out. So while customers can access the menu digitally, their choice may be sold out. To remedy this, the addition of a webcam to capture a current image of the pastry case and display it alongside the menu needed to be added to the project.

Adding to the Menu

I made a few overall design choices for this project to make it as robust and autonomous as possible since I didn't want to have anything that would require the baristas to fuss with the setup at all or have to learn Linux command line. I also wasn't sure how much network traffic this could potentially get so I wanted to keep it as simple as possible to prevent anything like their network being overloaded by a constant video stream while they are also handling their financial transactions.

This lead me to choose to set up the live feed of the pastry case to actually be a static image taken at the initial request of the web page, then a new one taken when the page is refreshed. Since the goal is just to get an idea of what pastries are still available at that moment in the day.

To add this functionality of capturing the webcam image upon each page request to the existing web server setup from my last project, I started by updating the backend web server Python code. The updated code is pulling the static image taken by the webcam using the OpenCV Python library and saving to the images folder in the static directory so the front end HTML can display it.

I found that certain browsers (Chrome and Safari) by default cache images with the same file name in an effort to reduce network traffic, thus refreshing the page wouldn't provide an updated image from the webcam. I'm not very strong with JavaScript/PHP to fix this issue from that perspective, so I simply used Python's random number generator function to append a random number to the image file name every time a new image is captured. This new image name is also passed to the HTML front end in the render_template() function so it knows which image file is the latest. This ensured that the image on the web page would always be up to date after a refresh.

I also added an os system function to remove all image files with the common image file name at the beginning of every page request to ensure the web server doesn't fill the memory of the MiniZed with outdated images.

Updated Python backend (webserver.py from the previous project post):

# MiniZed webserver script 
import os
import re
import cv2
import sys
import glob
import time
import uuid
import crypt
import signal
import random
import os.path
import subprocess
import multiprocessing  
from flask import Flask, render_template, request
from multiprocessing import Pipe

CUR_DIRECTORY = os.path.split(os.path.abspath(__file__))[0]
ALERT = -1

app = Flask(__name__)
app.debug = True

global timer_status, timeout, elapsed_time, pconn, cconn
timeout = 15
elapsed_time = 5
timer_status = "timer_disabled"
pconn, cconn = Pipe()

@app.route('/')
@app.route("/home.html", methods=["GET", "POST"])
def home():
    if request.method == "POST":
        os.system('rm /home/root/minized-webserver/static/images/single_frame*')

        randNum = str(random.randrange(0, 32000, 1))
        imageName = "single_frame"+randNum+".jpg"
        
        camera = cv2.VideoCapture(0)
        (success,reference) = camera.read()
        cv2.imwrite('/home/root/minized-webserver/static/images/'+imageName+'',reference)
        cv2.waitKey(1)
        cv2.VideoCapture.release(camera)
        cv2.waitKey(1)
        cv2.destroyAllWindows()
        cv2.waitKey(1)
        
        return render_template("Home/home.html", imageName=imageName)
    else:
        os.system('rm /home/root/minized-webserver/static/images/single_frame*')

        randNum = str(random.randrange(0, 32000, 1))
        imageName = "single_frame"+randNum+".jpg"
        
        camera = cv2.VideoCapture(0)
        (success,reference) = camera.read()
        cv2.imwrite('/home/root/minized-webserver/static/images/'+imageName+'',reference)
        cv2.waitKey(1)
        cv2.VideoCapture.release(camera)
        cv2.waitKey(1)
        cv2.destroyAllWindows()
        cv2.waitKey(1)
        
        return render_template("Home/home.html", imageName=imageName)

if __name__=='__main__':
    app.run(host='0.0.0.0', port=80, threaded=True)

Updated HTML front end of the homepage from the previous project post:

{% extends "Default/default.html" %}
{% block content %}

<div class="page-header">
    <h1 class="display-4"><b>{% block title %}Filling Station Menu{% endblock %}</b></h1>
</div>

<style>
img.one{
    height:100%;
    width:100%;
}
</style>

<img class="one" src="{{ url_for('static', filename='images/fs_window_menu-2.png') }}" width="1000" height="1000"/>

<p>(Refresh page to update image)</p>
<img class="one" src="/static/images/{{imageName}}" width="375" height="225"/>

{% endblock %}

At first, I had the feed from the USB camera and the menu on separate webpages with the menu being the home page, but then I realized it wasn't intuitive to check for the second page for the image from the webcam (especially since the whole point of the webcam is to show which items on the menu are still available that day). So I changed it to show all on the homepage with the menu on top and the image from the webcam below, with a prompt to inform the user that refreshing the page will update the image.

Modifying the Startup Script & Adding Reboot Button

Since it's not feasible to have a human operator manually connect to the coffee shop's wi-fi and launch the web server Python script at every boot up of the MiniZed, I knew I needed to add these tasks to the boot process of the embedded Linux image.

I also found that if the FPGA was power cycled using the Reset button on the MiniZed (SW2) or if it lost power without Linux having a chance to run the reboot or shutdown commands, Linux was unable to re-enumerate the wi-fi chip. This was obviously a huge issue since the FPGA could loser power any time due to a power outage at the coffee shop or which outlet the FPGA is plugged into needing to be moved. Overall, there needs to be a very simple, no-technical-knowledge-required method for properly power cycling the FPGA that doesn't involved using the Linux command line.

The best solution I came up with to fit this criteria was to have a push button on the MiniZed prompt the embedded Linux image to run the reboot command. Looking at the schematic of the MiniZed, I found that the PS Button (SW3) is connected the MIO of the Zynq chip making it readily available to configure and use in Linux.

A simple Python script running as a background task that is monitoring the PS Button' GPIO pin and makes an OS system call to reboot on logic level high of the GPIO pin did the job quite nicely.

Code to control the PS Button (SW3):

import os
import sys
import time
import subprocess
import multiprocessing

button_is_exported = os.path.exists('/sys/class/gpio/gpio881')

if button_is_exported == False:
    os.system('echo 881 > /sys/class/gpio/export')
    time.sleep(1)
    print('Exported PS button GPIO pin 881...')

def read_button():
    button = open('/sys/class/gpio/gpio881' + '/value', 'r')
    value = button.read()
    return value

while True:
    value = read_button()
    button_pressed = value.find('1')

    if button_pressed is 0:
        os.system('reboot')
        break
    else:
        continue

The script that runs during the boot up of the MiniZed is called minized-mount.sh and is in the ./project-spec/meta-user/recipes-bsp/minized-misc/files directory of the PetaLinux project when created from the MiniZed BSP.

I added to it the commands for connecting to the wi-fi network from the stored wpa_supplicant.conf file, followed by the launch of the flask web server and PS button control script as background tasks. I added a test with exit condition after each launch of the Python script to still allow the boot up to complete if one or both of the Python scripts failed to launch for some reason.

Updated minized-mount.sh script:

#!/bin/sh

# Add any custom code to be run at startup here

echo Running Minized Startup Script ...
echo
echo Mounting /dev/mmcblk1p1 on /mnt/emmc
mkdir /mnt/emmc
mount /dev/mmcblk1p1 /mnt/emmc
echo
echo Mounting /dev/sda1 on /mnt/usb
mkdir /mnt/usb
mount /dev/sda1 /mnt/usb
echo MiniZed Startup Script DONE!

echo Mount the eMMC.
mount /dev/mmcblk1p1 /mnt

echo Copy the wpa_supplicant.conf file from eMMC to /etc
cp -f /mnt/wpa_supplicant.conf /etc/.

echo Unmount the eMMC
umount /mnt


echo Bringup the WiFi interface...
wpa_supplicant -Dnl80211 -iwlan0 -c/etc/wpa_supplicant.conf -B


echo Fetch an IP address from the DHCP server...
udhcpc -i wlan0

FLASK_SERVER="/home/root/minized-webserver/webserver.py"
FLASK_CMD="python3 ${FLASK_SERVER}"

test -e "$FLASK_SERVER" || exit 0

echo Starting MiniZed Flask Webserver
start-stop-daemon --start --quiet --background --make-pidfile --pidfile /var/run/$FLASK_PID_NAME.pid --exec $FLASK_CMD

echo MiniZed Webserver is running!

PS_BUTTON="/home/root/ps_button_reboot.py"
PS_BUTTON_CMD="python3 ${PS_BUTTON}"

test -e "$PS_BUTTON" || exit 0

echo Enabling PS Button SW3
start-stop-daemon --start --quiet --background --make-pidfile --pidfile /var/run/$PS_BUTTON_PID_NAME.pid --exec $PS_BUTTON_CMD

echo PS Button SW3 is enabled!
exit 0

Testing the Setup

After some initial testing, I found that there was a stability issue with the webcam image capture continuously over a period of time. After a seemingly random number times(different every time), the webcam and MiniZed as a whole would freeze mid-capture of an image and require a hard system reboot (hitting the reset button SW2 on the MiniZed). This was also paired with the issue of the webcam constantly connecting and disconnecting itself upon reboot.

I thought maybe it was a bad USB cable on the webcam, so I tried a different webcam and saw the same issues. I then thought it might have been the USB cable I was using to supply the auxiliary power to the MiniZed since a lack of power is a common issue to cause peripherals to misbehave in this way. I found that the constant connecting/disconnecting lessened some when I switched the AUX power USB cable to a 5V/1A power brick from the USB port of my computer. However, after trying a few different USB cables and getting the same result, I ruled it out as well.

My last thought was that maybe I was still missing some drivers in the kernel and/or root filesystem. This suspicion was bolstered by the fact that when I used the lsusb command to list the USB devices seen by the kernel, it only could ID that the device contained Microdia hardware while the usb-devices command could only ID that the webcam contained Sonix hardware, but couldn't ID the specific chipset.

So I returned to the PetaLinux project to add what I had to guess would be the missing drivers. Starting in the kernel configuration editor:

petalinux-config -c kernel

Fixing the Kernel

Since the base chipset for the USB webcam I chose seems to be Sonix (as hinted by the output from the usb-devices command), I used the search function in the kernel configuration editor of PetaLinux (accessed anytime by pressing the "/" key) to see if there were any kernel drivers specifically to support Sonix, which turned out to be a few.

Under the path Device Drivers > Multimedia support > Media USB adapters > GSPCA based webcams, I enabled the following:

• SONIX Dual-Mode USB Camera Driver
• SN9C20X USB Camera Driver
• SONIX Bayer USB Camera Driver
• SONIX JPEG USB Camera Driver

It's also worth noting that these drivers are dependent on the USB_GSPCA and VIDEO_V4L2 kernel drivers which were enabled in my previous project post leading up to this one on Using a USB Web Camera with the MiniZed.

After applying these updates to the kernel and adding the root filesystem packages (listed below in the section Fixing the Root Filesystem), the first webcam I was using that was freezing/constantly connecting/disconnecting at boot up finally revealed that it wasn't compatible with the Video4Linux driver (after spending more time than I'd like to admit troubleshooting it with the v4l2-ctl commands). So I switched to a different one I grabbed off Amazon.

While it didn't freeze after capturing an image like the first one had, I was still plagued by it constantly connecting and disconnecting at boot up. After I would physically disconnect the USB and plug it back in a few times, it would finally stay connected as normal. But this obviously wasn't an acceptable to leave and force the baristas to deal with in the event the FPGA lost power ever and reset the USB bus.

I ran the lsusb command to see what the system read the webcam as on the bus now:

Googling Novatel Wireless Merlin U740 led me its kernel driver page on linux-hardware.org where I noticed it also required a serial interface conversion driver that I didn't remember ever adding to my kernel in the past.

Using the search option in the PetaLinux kernel configuration editor, I found the CONFIG_USB_SERIAL and CONFIG_USB_SERIAL_OPTION drivers that I was missing:

Under Device Drivers >USB Support, I enabled USB Serial Converter support which became a new tab with more kernel driver options. Within USB Serial Converter support I enabled USB driver for GSM and CDMA modems.

Alas, my problems were still not over. After adding the USB serial converter drivers to the kernel, the constant connecting/disconnecting of the USB device intensified to the point that there was no amount of physical unplugging/plugging back in of the webcam USB plug that could make it stop. My instinct was still telling me this was a power issue because I would see the webcam connect as a USB device, fully enumerate, but then immediately disconnect. At this point, it was clear I still had something not configured right in my kernel.

Because I was still convinced the behavior was power related, I started looking at any USB kernels that had to do with power control of the bus. I spent some time digging through all of the USB kernel options in the kernel configuration editor, and using the < Help > option on anything that seemed to be power related under Device Drivers > USB support. The < Help > option in PetaLinux provides a brief description of each driver option, and in many cases will advise you whether or not to enable to disable certain options.

The first thing I noticed was the Maximum VBUS Power usage was set to limit the USB port to provide no more than 2mA. The typical webcam draws around 150mA during operation, so the default 2mA limit in the kernel was obviously contributing somewhat to the issue. Both webcams I had tried thus far were rated for 500mA peak power, so I changed the Maximum VBUS Power usage to the max of 500mA.

The other main kernel driver I found that heavily involved controlling the power of the USB bus was the USB OTG driver. USB On-The-Go protocol allows for USB peripherals to dynamically switch between being the host (master) and the device (slave), and the host device on the USB bus is responsible for providing the power to the bus (VBUS) for operation.

This was the root culprit to my problem, something about the webcams was confusing the OTG driver of the MiniZed into thinking they were hosts instead of devices, so the MiniZed would shut off the power going to VBUS thinking that the webcams were going to power it. Since the webcams have no way of providing power to the VBUS, it just ended up in this loop of constant power cycling which was the cause of the constant connecting/disconnecting.

Both the kernel driver for OTG was enabled and the drive mode in the device tree for the USB's node was set to OTG in the BSP image of the MiniZed, so I changed the drive mode setting in the MiniZed's device tree to host and disabled OTG support in the kernel under Device Drivers > USB support.

After disabling OTG and increasing the max VBUS current limit, the constant connecting/disconnecting issue disappeared and the USB bus was very stable. Unfortunately I found that the second webcam I bought with the USB serial converter still had intermittent issues with the Video4Linux driver. So I switched out webcams again to one that I found on the list of USB video class devices (a Logitech HD Webcam C270h AP) so I knew it would have no issues with the current Video4Linux and UVC drivers.

While this journey consumed more time than I'd like to admit, I do feel it's given me a better intuitive understanding of the dynamic between the embedded Linux kernel and various types of hardware. Especially with how hardware can still work with certain drivers missing, but just demonstrate weird and unexpected behavior.

I know this whole explanation of my troubleshooting is long-winded and verging on irrelevant, but the demonstration of how to troubleshoot/discover issues I hope can be helpful (at least I've found them helpful in other people's blog posts, I'd love feedback on what you think as a reader in the comments!).

Fixing the Root Filesystem

Moving on to the root filesystem configuration editor to add the missing library packages:

petalinux-config -c rootfs

I found that the libusb and libudev packages had not been enabled, which are necessary packages for making USB device functions available to the user space from the kernel.

Under the path Filesystem Packages > misc > eudev, I enabled the following

• libudev
• udev-extraconf (was already enabled)

Under Filesystem Packages > libs > gtk+, I enabled gtk+ and in Filesystem Packages > libs > libusb-compat, I enabled libusb-compat. Finally, under Filesystem Packages > libs > libusb1, I enabled libusb1.

After exiting the configuration editor, I rebuilt the project and generated a new boot binary file

petalinux-build
petalinux-package --boot --fsbl <project directory>/images/linux/zynq_fsbl.elf --fpga <project directory>/images/linux/system.bit --u-boot --force

After updating the MiniZed with the new image, I re-installed the Python packages Flask and Requests:

root@minized-emmc-enhanced-2019-2:~# pip3 install flask --trusted-host pypi.org --trusted-host files.pythonhosted.org

root@minized-emmc-enhanced-2019-2:~# pip3 install requests --trusted-host pypi.org --trusted-host files.pythonhosted.org

Enclosure for FPGA that also Supports Webcam

Given that I don't own a 3D printer, I looked for a simple off-the-shelf options for an enclosure for the MiniZed. I also wanted it to also serve as a way for mounting the webcam. Surprisingly, I found that a plastic sandwich box was a great solution for the MiniZed.

I cut a hole in the bottom half of the box similar to one on a picture frame where it's a large hole with a small notch at the top to allow for it to be hung on a hook/nail on a wall. A second hold in the wall of the box for the USB cables (for power, UART, & the webcam) to pass through.

At first I just used a dot a super glue to each of the MiniZed's rubber feet to mount it in the sandwich box. However I was worried it wouldn't hold up to the force applied with plugging/unplugging either of the USB cables. I also realized that the hole in the bottom needed to be larger to pass the USB cables through.

Some simple mirror holder brackets worked nicely to hold the MiniZed in place, and gaffer's tape worked to protect the cables from the raw cut edges of the plastic.

I decided to put my Dymo labeler to work using it to write the instructions on how/when to use the PS Button to reboot the MiniZed in the event that something isn't working right.

Finally to mount the webcam, a strip of velcro on the front side of the lid and the foot of the webcam did the trick.

Installing the FPGA in the Coffee Shop

I have been been connected with the manager of my local coffee shop on Facebook for a while (just to give you an idea of how much I've frequented this coffee shop) where he had posted that he was looking for a digital menu solution after the pandemic hit. This is what started this whole project, and finally getting to install it in the coffee shop was a great feeling.

The to major things that needed to be done in the installation were:

1 - Connect the MiniZed to the coffee shop's network and open a port for it (which would provide the link to embed in a QR code).

2 - Determine the best position for the webcam to see the pastry case.

Luckily their pastry case's shelves and sides are clear glass so the webcam could be easily mounted close on the wall and the pastries on the top shelf arranged such that one could see the pastries on the bottom shelf through the clear glass.

I brought a few different options for hanging the box on the wall including command strips, nails, and thumb tacks since I wasn't sure exact where hanging the whole setup would end up being the best. A small nail was perfect for the wood trim where I ended up mounting the box.

Enabling Port Forwarding for External Network Access

To connect the MiniZed to the coffee shop's network and open a port for it, I partnered with their IT team to create a secondary network for it that was separated from the network handling the shop's POS transactions. I also disabled SSH on the MiniZed since putting it on an open port would leave the potential for anyone to log into it otherwise.

To disable SSH, I opened the SSH configuration file in a text editor:

root@minized-emmc-enhanced-2019-2:~# vi /etc/ssh/sshd_config

I commented out all of the rules for the host and added the following line to disable root login:

PermitRootLogin no

I updated the wpa_supplicant.conf file stored on the MiniZed (under /run/media/mmcblk1p1) with the new wifi network's ESSID and password. The router was configured to give the MiniZed a DHCP static lease given that maintaining a static IP is necessary for the port forwarding configuration on the router.

For the port forwarding setup, the MiniZed's Flask server is running on port 80 since the incoming requests are HTTP requests. Thus both the internal and external ports in the forwarding configuration will be on port 80.

The app.run() function in Python is where the desired port for the server is passed (from webserver.py):

if __name__=='__main__':
    app.run(host='0.0.0.0', port=80, threaded=True)

Since the external port is port 80, the link for the QR code becomes the router's external IP with the port specified:

<router's external ip address>:80

Generating the QR Code for Customers to Use

There are many sites that can be used to generate a simple QR code for free like this one. Using the link from the port forwarding setup, a.PNG file of the new QR code is generated and made available for download.

This QR code will be printed and placed on the wall of the building at the beginning of the drive through.

Summary

Overall this project was a lot of fun and very satisfying to see in action. It also taught me a lot as far as how to create a full solution from my FPGA designs with features such as the PS Button for reboot and adding the startup of applications to the boot process of the Zynq chip.

There are still a few more features I'd like to add such as an automated script for the baristas to upload a new menu via the UART serial port. I'm sure this will be a constantly evolving project and I'm exited to see what it turns into.