Using a USB Web Camera with the MiniZed

I recently started using a USB web camera with my Ultra96v2, this lead to me wanting to implement a camera into my other FPGA designs starting with the MiniZed. I'm stuck on the MiniZed because I really feel that it doesn't get the credit it deserves. It's small form factor, with 8GB on eMMC storage, and minimal power requirements make it super low maintenance. It's an ideal board for bench top setups that need to be minimal (which we all can appreciate in this era of working from home), as well as "set it and forget it" type projects. The driver for the wireless chip is very stable and I've found myself naturally gravitating to it simply because I don't have to tweak it every time I use it in a new PetaLinux project.

Upon plugging in my spare web camera, I quickly found that the BSP (board support package) for the MiniZed did not have the necessary drivers enabled by default for using a USB web camera. After some trial and error, I was able to figure out what drivers and packages were needed.

For reference: I am starting with the ext4 file system image I created for the MiniZed in one of my past projects here. I am using PetaLinux version 2019.2 again here too.

Update the Kernel

The MiniZed's kernel needs the USB gadget drivers, web camera/video drivers, Xilinx DRM KMS driver, and frame buffer driver enabled/added in order to support the OpenCV library and communicating with the web camera to transfer images.

Launch the kernel configuration editor GUI:

petalinux-config -c kernel

Enable the following packages (you can search for each by typing "/" at any point when the GUI is active in the terminal window):

CONFIG_MEDIA_CAMERA_SUPPORT
CONFIG_MEDIA_CONTROLLER
CONFIG_VIDEO_V4L2_SUBDEV_API
CONFIG_VIDEO_ADV_DEBUG
CONFIG_MEDIA_USB_SUPPORT
CONFIG_USB_VIDEO_CLASS
CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV
CONFIG_USB_GSPCA
CONFIG_V4L_PLATFORM_DRIVERS
CONFIG_VIDEO_XILINX
CONFIG_VIDEO_XILINX_AXI4S_SWITCH
CONFIG_VIDEO_XILINX_CFA
CONFIG_VIDEO_XILINX_CRESAMPLE
CONFIG_VIDEO_XILINX_DEMOSAIC
CONFIG_VIDEO_XILINX_GAMMA
CONFIG_VIDEO_XILINX_HLS
CONFIG_VIDEO_XILINX_REMAPPER
CONFIG_VIDEO_XILINX_RGB2YUV
CONFIG_VIDEO_XILINX_SCALER
CONFIG_VIDEO_XILINX_MULTISCALER
CONFIG_VIDEO_XILINX_SDIRXSS
CONFIG_VIDEO_XILINX_SWITCH
CONFIG_VIDEO_XILINX_TPG
CONFIG_VIDEO_XILINX_VPSS_CSC
CONFIG_VIDEO_XILINX_VPSS_SCALER
CONFIG_VIDEO_XILINX_VTC
CONFIG_VIDEO_XILINX_CSI2RXSS
CONFIG_VIDEO_XILINX_SCD
CONFIG_VIDEO_XILINX_M2M
CONFIG_DRM_XILINX
CONFIG_DRM_XILINX_DP
CONFIG_DRM_XILINX_DP_SUB
CONFIG_DRM_XLNX
CONFIG_DRM_XLNX_BRIDGE
CONFIG_DRM_XLNX_DSI
CONFIG_DRM_XLNX_MIXER
CONFIG_DRM_XLNX_PL_DISP
CONFIG_DRM_XLNX_SDI
CONFIG_DRM_XLNX_BRIDGE_CSC
CONFIG_DRM_XLNX_BRIDGE_SCALER
CONFIG_DRM_XLNX_BRIDGE_VTC
CONFIG_FB_XILINX
CONFIG_USB
CONFIG_USB_PCI
CONFIG_USB_ANNOUNCE_NEW_DEVICES
CONFIG_USB_OTG
CONFIG_USB_OTG_FSM
CONFIG_USB_EHCI_HCD
CONFIG_USB_EHCI_ROOT_HUB_TT
CONFIG_USB_EHCI_TT_NEWSCHED
CONFIG_USB_STORAGE
CONFIG_USB_UAS
CONFIG_USB_CHIPIDEA
CONFIG_USB_CHIPIDEA_UDC
CONFIG_USB_CHIPIDEA_HOST
CONFIG_USB_GADGET
CONFIG_USB_GADGET_XILINX
CONFIG_USB_CONFIGFS
CONFIG_USB_CONFIGFS_SERIAL
CONFIG_USB_CONFIGFS_ACM
CONFIG_USB_CONFIGFS_OBEX
CONFIG_USB_CONFIGFS_NCM
CONFIG_USB_CONFIGFS_ECM
CONFIG_USB_CONFIGFS_ECM_SUBSET
CONFIG_USB_CONFIGFS_RNDIS
CONFIG_USB_CONFIGFS_EEM
CONFIG_USB_CONFIGFS_MASS_STORAGE
CONFIG_USB_CONFIGFS_F_LB_SS
CONFIG_USB_CONFIGFS_F_FS
CONFIG_USB_CONFIGFS_F_UAC1
CONFIG_USB_CONFIGFS_F_UAC1_LEGACY
CONFIG_USB_CONFIGFS_F_UAC2
CONFIG_USB_CONFIGFS_F_MIDI
CONFIG_USB_CONFIGFS_F_HID
CONFIG_USB_CONFIGFS_F_UVC
CONFIG_USB_CONFIGFS_F_PRINTER
CONFIG_USB_GADGETFS

Save and exit the kernel configuration editor after enabling the necessary drivers.

Update the Root Filesystem

The root filesystem needs several packages added both though the configuration editor GUI and the config file in the meta-user directory of the PetaLinux project. This includes the the OpenCV libraries, Video4Linux libraries, GStreamer framework, and X11 framework.

Before launching the configuration GUIs, add the following to the root filesystem configuration file in <PetaLinux project directory>/project-spec/meta-user/conf/user-rootfsconfig This makes the configuration editor GUI aware that the specified packages are an option to add to the build.

# Local to project
CONFIG_peekpoke
CONFIG_gpio-demo

CONFIG_packagegroup-base-extended
CONFIG_packagegroup-petalinux-gstreamer

# Minized specific customizations
CONFIG_i2csensor
CONFIG_dialog-control
CONFIG_get-gpio-offsets
CONFIG_minized-misc
CONFIG_minized-wireless

# LIS2DS12 Accelerometer driver module
CONFIG_lis2ds

# Extra packages
CONFIG_gsl
CONFIG_cmake
CONFIG_iperf3
CONFIG_nano
CONFIG_hostapd
CONFIG_iw
CONFIG_bluez5-doc
CONFIG_pulseaudio-module-bluetooth-discover
CONFIG_pulseaudio-module-bluetooth-policy
CONFIG_pulseaudio-module-bluez5-discover
CONFIG_pulseaudio-module-bluez5-device
CONFIG_pulseaudio-module-switch-on-connect

# Python3
CONFIG_python3
CONFIG_python3-pip
CONFIG_python3-cffi
CONFIG_python3-numpy
CONFIG_python3-shell
CONFIG_python3-pyserial
CONFIG_python3-threading
CONFIG_python3-multiprocessing

# OpenCV packages
CONFIG_packagegroup-petalinux-opencv
CONFIG_packagegroup-petalinux-x11
CONFIG_packagegroup-petalinux-v4lutils

# Custom Knitronics recipe files
CONFIG_gameboard
CONFIG_webserver

Launch the root filesystem configuration editor GUI:

petalinux-config -c rootfs

Enable the following Filesystem Packages for the Gstreamer library needed to support the Python OpenCV (cv2) library:

CONFIG_gstreamer1.0
CONFIG_gstreamer1.0-plugins-bad
CONFIG_gstreamer1.0-plugins-base
CONFIG_gstreamer1.0-plugins-good

Under the Petalinux Package Groups tab, enable the package-petalinux, package-petalinux-display-debug, package-petalinux-opencv, package-petalinux-self-hosted, package-petalinux-utils, package-petalinux-v4lutils, and package-petalinux-x11 options.

Then navigate to the user packages tab to enable all of the custom options that were added to the user-rootfsconfig file:

Save and exit the root filesystem configuration editor then build the PetaLinux project:

petalinux-build

You may seen a warning about the kernel being "tainted from a forced run", but you can safely ignore this warning. It's just a quirk about 2019.2 where it complains anytime you modify the kernel settings of a project.

Package New Boot Binary Image

After the PetaLinux project has successfully finished building, package the new boot binary image for the MiniZed containing the the FSBL for the Zynq, the bitstream for the programmable logic, and the u-boot environment.

petalinux-package --boot --fsbl <project directory>/images/linux/zynq_fsbl.elf --fpga . <project directory>/images/linux/system.bit --u-boot --force

Upload New Image to MiniZed

To upload a new root filesystem to the MiniZed that's an ext4 type booted from the eMMC of the MiniZed, we first have to revert back to the initramfs root filesystem so that the ext4 partition of the eMMC is not locked by the filesystem running on it.

Since this PetaLinux project was built using the eMMC boot BSP for the MiniZed, the kernel and boot binary for the initramfs root filesystem image can be found in the pre-built directory of the project. The kernel is image.ub and the initramfs boot binary is BOOT_EMMC.BIN

Using a USB flash drive (you will need a second USB cable plugged into the AUX Power micro-USB port of the MiniZed to provide power to its USB Type A port), copy the initramfs kernel & BOOT_EMMC.BIN to /run/media/mmcblk1p1 and write the BOOT_EMMC.BIN to the MiniZed's QSPI flash. Then reboot the board into the initramfs system.

root@MiniZed:~# cp /run/media/sda1/BOOT_EMMC.BIN /run/media/mmcblk1p1
root@MiniZed:~# cp /run/media/sda1/image.ub /run/media/mmcblk1p1
root@MiniZed:~# flashcp /run/media/mmcblk1p1/BOOT.BIN /dev/mtd0
root@MiniZed:~# reboot

With your modified wpa_supplicant.conf still living in the /run/media/mmcblk1p1 directory, run the wi-fi setup script to connect the MiniZed to your wireless network:

root@MiniZed:~# wifi.sh

As I outlined previously, the initramfs image doesn't have the necessary libraries for handling ext4 so we have to copy them over manually (I covered how to generate these library files in a past project here). I found that using a secure copy straight from my PC to the MiniZed is the best method of doing this:

scp -p ./lib/libcom_err.so     root@192.168.1.xxx:/lib
scp -p ./lib/libcom_err.so.2   root@192.168.1.xxx:/lib
scp -p ./lib/libcom_err.so.2.1 root@192.168.1.xxx:/lib
scp -p ./lib/libe2p.so         root@192.168.1.xxx:/lib
scp -p ./lib/libe2p.so.2       root@192.168.1.xxx:/lib
scp -p ./lib/libe2p.so.2.3     root@192.168.1.xxx:/lib
scp -p ./lib/libext2fs.so      root@192.168.1.xxx:/lib
scp -p ./lib/libext2fs.so.2    root@192.168.1.xxx:/lib
scp -p ./lib/libext2fs.so.2.4  root@192.168.1.xxx:/lib

scp -p ./sbin/mke2fs.e2fsprogs root@192.168.1.xxx:/sbin
scp -p ./sbin/mkfs.ext4        root@192.168.1.xxx:/sbin

After the eMMC is partitioned & formatted the first time out of the box, you don't need to do it again unless it gets corrupted. In my case, since I'm updating the ext4 filesystem, I don't need to reformat the partition of the eMMC it's on (/dev/mmcblk1p2). This means I can skip straight to mounting it to a local directory and erasing the old ext4 root filesystem on it (if you haven't previously formatted the MiniZed's eMMC, follow the first part of my past project here).

root@MiniZed:~# mkdir /mnt/emmc_rootfs
root@MiniZed:~# mount /dev/mmcblk1p2 /mnt/emmc_rootfs
root@MiniZed:~# rm -r /mnt/emmc_rootfs/*

Copy over the new rootfs.ext4 from the USB flash drive and mount a temporary drive to the second partition to extract it into for the extra space. Copy the root filesystem files back into the main drive of the eMMC's second partition then unmount and delete the other temporary drive directory.

root@MiniZed:~# cp /run/media/sda1/<new_linux>/rootfs.ext4 /mnt/emmc_rootfs
root@MiniZed:~# mkdir /mnt/ext4
root@MiniZed:~# mount /mnt/emmc_rootfs/rootfs.ext4 /mnt/ext4/ -o loop
root@MiniZed:~# cp -rf /mnt/ext4/* /mnt/emmc_rootfs/
root@MiniZed:~# umount /mnt/ext4/
root@MiniZed:~# rm /mnt/emmc_rootfs/rootfs.ext4

Copy the new kernel and boot binary for the new ext4 root filesystem to the first partition of the eMMC and flash the QSPI with the new boot binary.

root@MiniZed:~# cp /run/media/sda1/<new_linux>/image.ub /run/media/mmcblk1p1
root@MiniZed:~# cp /run/media/sda1/<new_linux>/BOOT.BIN /run/media/mmcblk1p1
root@MiniZed:~# flashcp /run/media/mmcblk1p1/BOOT.BIN /dev/mtd0

Finally, fully power cycle the board by sending the shutdown command then using the Reset button (SW2) on the board to power it back on:

root@MiniZed:~# shutdown -h now

Test the USB Web Camera

With the second USB cable still plugged into the AUX Power micro-USB port of the MiniZed to provide power to its USB Type A port, plug in the web camera.

Use the usb-devices command to list all of the USB devices the kernel currently sees in the hardware and which drivers it is tying the hardware too. You'll see the web camera appear and the USB gadget drivers assigned to it:

You can verify the Video4Linux driver recognizes the device using v4l2-ctl and listing the current devices available.

The OpenCV library is available in Python3, so a simple test script can be written in Python to capture an image and write it to the home directory of the MiniZed.

import os
import cv2
import time 

capture = cv2.VideoCapture(0)
time.sleep(0.1)
(success, reference) = capture.read()

cv2.imwrite('/home/root/test_image.jpg',reference)

capture.release()
cv2.destroyAllWindows()

To check the image, I connected the MiniZed to my wireless network and used the secure copy (scp) command from the command line of my computer to transfer it to my local drive.

scp -p root@191.168.1.xxx:/home/root/test_image.jpg /<my local directory>

And voila! The image!

Now my MiniZed is photo/video capable and ready for a new project!