Hardware components | ||||||
 |
| × | 1 | |||
 |
| × | 1 | |||
Software apps and online services | ||||||
 |
| |||||
 |
| |||||
This project is part 1 of a 4 part series of projects, where we will progressively create a Vitis-AI and ROS2 enabled platform for Ultra96-V2:
- Part 1 : Building the foundational designs
- Part 2 : Combining designs into a common platform
- Part 3 : Adding support for Vitis-AI
- Part 4 : Adding support for ROS2
The motivation of this series of projects is to enable users to create their own custom AI applications.
Introduction - Part IEvery time Avnet releases a new petalinux BSP version for its platforms, it's like Christmas.
I always start my designs from these BSPs, which include both hardware (Vivado) and software (Petalinux) components.
Foundational Design OverviewI call these projects the foundational designs, since they are literally the foundation on which I create more advanced projects, including hardware accelerators and additional software packages.
The Ultra96-V2 has two (2) foundational designs:
- u96v2_sbc_base
- u96v2_sbc_dualcam
The first design, the base design, is the simplest design. It has an (almost) empty PL design.
The second design, the dualcam design, implements a MIPI capture pipeline in the PL (hardware), and supports the V4L2 API in linux (software).
The next sections describe how to rebuild these foundational designs from source in the Avnet github repositories.
Cloning the Avnet github repositoriesThe foundational designs make use of the following Avnet github repositories:
- github.com/Avnet/bdf (latest branch)
- github.com/Avnet/hdl (2022.2 branch)
- github.com/Avnet/petalinux (2022.2 branch)
- github.com/Avnet/meta-avnet (2022.2 branch)
- github.com/Avnet/meta-on-semiconductor (2022.2 branch)
The first thing to do is to clone the three (3) first repositories to a Ubuntu linux machine which has the 2022.2 tools installed. I like to do this in a directory that has a meaningful name (such as "Avnet_2022_2" in your root directory):
$ cd ~
$ mkdir Avnet_2022_2
$ cd ~/Avnet_2022_2
$ git clone https://github.com/Avnet/bdf
$ git clone -b 2022.2 https://github.com/Avnet/hdl
$ git clone -b 2022.2 https://github.com/Avnet/petalinuxThe last two repositories will get cloned during the build process, so no need to clone them.
Building u96v2_sbc_baseThe first design we will build is the "u96v2_sbc_ base" design.
To do this, first call your 2022.2 setup scripts:
$ source /tools/Xilinx/Vitis/2022.2/settings64.sh
$ source /tools/Xilinx/petalinux-v2022.2-final/settings.shNext, launch the petalinux build script as follows:
$ cd ~/Avnet_2022_2/petalinux
$ scripts/make_u96v2_sbc_base.shThis script will perform the following:
- Create and Build the Vivado project
~/Avnet_2022_2/hdl/projects/u96v2_sbc_base_2022_2 - Create and Build the Petalinux project
~/Avnet_2022_2/petalinux/projects/u96v2_sbc_base_2022_2 - Create a final SD card image
~/Avnet_2022_2/petalinux/projects/u96v2_sbc_base_2022_2/images/linux/rootfs.wic
I often get an error during the petalinux build phase, caused by a failure to access packages from various github repositories. If this occurs, re-check your internet connection, and re-start the build as follows:
$ cd ~/Avnet_2022_2/petalinux/u96v2_sbc_base_2022_2
$ petalinux-buildI usually keep doing this until the build succeeds.
For the Ultra96-V2, I often am not able to build the following openamp-fw-* packages:
- openamp-fw-echo-testd
- openamp-fw-mat-muld
- openamp-fw-rpc-demo
In order to workaround this issue, I simply remove them from the petalinux-image definition. To do this, edit the following file: ~/Avnet_2022_2/petalinux/u96v2_sbc_base_2022_2/roject-spec/meta-avnet/recipes-core/images/petalinux-image-minimal.bbappend
Remove the "openamp-fw-*" packages from the "IMAGE_INSTALL:append:u96v2-sbc" entry:
IMAGE_INSTALL:append:u96v2-sbc = "\
...
"It is important to know that the automated petalinux build configured the petalinux project to cache all the packages that are fetched from the internet. That way, those packages are only fetched once for all the petalinux projects, and reused from the local cache.
If your automated build succeeded, the cache will have been disabled in the petalinux project. You "may" want to re-enable it by editing the following files:
project-spec/configs/config
...
#
# Add pre-mirror URL
#
CONFIG_PRE_MIRROR_URL="file://{path}/Avnet_2022_2/petalinux/projects/cache/downloads_2022.2/"
...
#
# Network sstate feeds URL
#
CONFIG_YOCTO_LOCAL_SSTATE_FEEDS_URL="file://{path}/Avnet_2022_2/petalinux/projects/cache/sstate_2022.2/aarch64/"
CONFIG_NETWORK_LOCAL_SSTATE_FEEDS=y
...project-spec/meta-user/conf/petalinuxbsp.conf
...
PREMIRRORS:prepend = "git://.*/.* file://{path}/Avnet_2022_2/petalinux/projects/cache/downloads_2022.2/ \
ftp://.*/.* file://{path}/Avnet_2022_2/petalinux/projects/cache/downloads_2022.2/ \
http://.*/.* file://{path}/Avnet_2022_2/petalinux/projects/cache/downloads_2022.2/ \
https://.*/.* file://{path}/Avnet_2022_2/petalinux/projects/cache/downloads_2022.2/ "
DL_DIR="{path}/Avnet_2022_2/petalinux/projects/cache/downloads_2022.2/"
SSTATE_DIR"{path}/Avnet_2022_2/petalinux/projects/cache/sstate_2022.2/"
...where {path} is the absolute path to your Avnet_2022_2 directory.
Executing u96v2_sbc_baseIn order to execute the base design, we first need to program the SD card image to a micro-SD card (of size 32GB or greater).
~/Avnet_2022_2/petalinux/projects/u96v2_sbc_base_2022_2/images/linux/rootfs.wic
To do this, we use Balena Etcher, which is available for most operating systems.
Once programmed, insert the micro-SD card into the Ultra96-V2, and connect up the platform as shown below.
Press the power push-button to boot the board.
As linux boots, a large quantity of verbose output will be sent to the serial console, ending with the following:
...
[ OK ] Started Start ultra96 python flask server.
[ OK ] Finished Delete wpa files in /var/run/wpa_supplicant.
[ OK ] Started Xserver startup without a display manager.
Starting OpenSSH Key Generation...
[ OK ] Finished IPv6 Packet Filtering Framework.
[ OK ] Finished IPv4 Packet Filtering Framework.
[ OK ] Started rng-tools.service.
[ OK ] Finished Resets System Activity Logs.
[ OK ] Finished Common GPIO Utilities.
[ OK ] Started Telephony service.
[ OK ] Reached target Preparation for Network.
Starting Connection service...
Starting inetd.busybox.service...
Starting Load Default Camera Configuration...
Starting Network Configuration...
Starting Ultra96 Radio (Bluetooth & WiFi) LEDs...
[ OK ] Started inetd.busybox.service.
Starting Load/Save RF Kill Switch Status...
[ OK ] Finished OpenSSH Key Generation.
[ OK ] Started Load/Save RF Kill Switch Status.
[ OK ] Found device /sys/subsystem/net/devices/wlan0.
[ OK ] Started User Login Management.
[ OK ] Started WPA supplicant dae… (interface-specific version).
[ OK ] Finished Ultra96 Radio (Bluetooth & WiFi) LEDs.
[ OK ] Started Network Configuration.
[ OK ] Started Connection service.
[ OK ] Reached target Network.
Starting Avahi mDNS/DNS-SD Stack...
[ OK ] Started NFS status monitor for NFSv2/3 locking..
Starting Network Time Service...
Starting Permit User Sessions...
Starting Target Communication Framework agent...
[ OK ] Started Xinetd A Powerful Replacement For Inetd.
[ OK ] Finished Permit User Sessions.
[ OK ] Started Avahi mDNS/DNS-SD Stack.
[ OK ] Started Getty on tty1.
[ OK ] Started Serial Getty on ttyPS0.
[ OK ] Reached target Login Prompts.
[ OK ] Finished Load Default Camera Configuration.
[ OK ] Started Target Communication Framework agent.
[ OK ] Started Network Time Service.
Starting Hostname Service...
[ OK ] Started Hostname Service.
PetaLinux 2022.2_release_S10071807 u96v2-sbc-dualcam-2022-2 ttyPS0Login as the "root" user as follows:
u96v2-sbc-base-2022-2 login: root
root@u96v2-sbc-base-2022-2:~#We will be using a DisplayPort monitor to view video output from the Ultra96-V2. You should observe the matchbox desktop on the DisplayPort monitor.
With a USB keyboard & mouse, click on "Connman Settings" to configure the WIFI.
Click on "Wireless", then select your WIFI network, and specify your passphrase.
Once the WIFI is connected, you can discover which IP address has been assigned to your Ultra96-V2 with the following command:
root@u96v2-sbc-base-2022-2:~# ifconfig -a wlan0
wlan0 Link encap:Ethernet HWaddr F8:F0:05:C4:17:66
inet addr:10.0.0.179 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::faf0:5ff:fec4:1766/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:61 errors:0 dropped:0 overruns:0 frame:0
TX packets:79 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7330 (7.1 KiB) TX bytes:12263 (11.9 KiB)
root@u96v2-sbc-base-2022-2:~#Next we can perform a quick sanity check with the USB camera.
We can query the presence of the USB camera with the following commands:
root@u96v2-sbc-base-2022-2:~# ls /dev/media*
/dev/media0
root@u96v2-sbc-base-2022-2:~# ls /dev/video*
/dev/video0 /dev/video1
root@u96v2-sbc-base-2022-2:~# media-ctl -p -d /dev/media0
Media controller API version 5.15.36
Media device information
------------------------
driver uvcvideo
model UVC Camera (046d:0825)
serial 33E07BA0
bus info usb-xhci-hcd.2.auto-1
hw revision 0x12
driver version 5.15.36
Device topology
...
root@u96v2-sbc-base-2022-2:~# v4l2-ctl -D -d /dev/video0
Driver Info:
Driver name : uvcvideo
Card type : UVC Camera (046d:0825)
Bus info : usb-xhci-hcd.2.auto-1
Driver version : 5.15.36
Capabilities : 0x84a00001
Video Capture
Metadata Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04200001
Video Capture
Streaming
Extended Pix Format
...
root@u96v2-sbc-base-2022-2:~# v4l2-ctl -D -d /dev/video1
Driver Info:
Driver name : uvcvideo
Card type : UVC Camera (046d:0825)
Bus info : usb-xhci-hcd.2.auto-1
Driver version : 5.15.36
Capabilities : 0x84a00001
Video Capture
Metadata Capture
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04a00000
Metadata Capture
Streaming
Extended Pix Format
...
root@u96v2-sbc-base-2022-2:~#We observe that one media node (/dev/media0) and two video nodes (/dev/video0, /dev/video1) have been enumerated.
Inspecting the "Device Caps" of each video node, we take note that the /dev/video0 is the one which has "Video Capture" capability.
Create a new python script in the root directory with the following content:
import cv2
# Open the camera
cap = cv2.VideoCapture(0)
# Set the resolution
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
# Create a display window
cv2.namedWindow("USB Cam Passthrough", cv2.WINDOW_NORMAL)
while True:
# Read a frame from the camera
ret, frame = cap.read()
if ret:
# Display the frame in the window
cv2.imshow("USB Cam Passthrough", frame)
# Check for key presses
key = cv2.waitKey(1)
if key == 27: # Pressing the Esc key exits the program
break
# Release the camera and destroy the display window
cap.release()
cv2.destroyAllWindows()NOTE : This code was generated by ChatGPT with the following prompt : "python code usbcam passthrough 640x480"
Run this script from a desktop terminal as follows:
root@u96v2-sbc-base-2022-2:~# python3 usbcam_passthrough.py
[ WARN:0] global /usr/src/debug/opencv/4.5.2-r0/git/modules/videoio/src/cap_gstreamer.cpp (1081) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1This will display the video from the USB camera in a window on the embeded desktop running the the Ultra96-V2.
In order to get a better understanding of the base design, we will modify it. More specifically, we will include additional software packages in the petalinux project.
AMD-Xilinx often provides groups of packages that are named with the "packagegroup-" prefix. As an example, we will add jupyter notebook functionality, which is provided by "packagegroup-petalinux-jupyter".
Edit the following file: ~/Avnet_2022_2/petalinux/u96v2_sbc_base_2022_2/roject-spec/meta-avnet/recipes-core/images/petalinux-image-minimal.bbappend
Add "packagegroup-petalinux-jupyter" to the "IMAGE_INSTALL:append:u96v2-sbc" entry:
IMAGE_INSTALL:append:u96v2-sbc = "\
...
packagegroup-petalinux-jupyter \
"Next, rebuild the petalinux project:
$ cd ~/Avnet_2022_2/petalinux/u96v2_sbc_base_2022_2
$ petalinux-buildOnce complete, reprogram the micro-SD card, and reboot the Ultra96-V2.
Once booted, launch the jupyter-lab server in the serial console, specifying the IP address, as follows:
petalinux@u96v2-sbc-dualcam-2022-2:$ jupyter-lab --ip 10.0.0.179 &
[1] 879
[I 2023-03-20 16:07:40.475 ServerApp] jupyterlab | extension was successfully linked.
[I 2023-03-20 16:07:40.548 ServerApp] Writing Jupyter server cookie secret to /home/petalinux/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2023-03-20 16:07:40.649 LabApp] JupyterLab extension loaded from /usr/lib/python3.9/site-packages/jupyterlab
[I 2023-03-20 16:07:40.649 LabApp] JupyterLab application directory is /usr/share/jupyter/lab
[I 2023-03-20 16:07:40.681 ServerApp] jupyterlab | extension was successfully loaded.
[I 2023-03-20 16:07:40.686 ServerApp] Serving notebooks from local directory: /home/root
[I 2023-03-20 16:07:40.686 ServerApp] Jupyter Server 1.13.5 is running at:
[I 2023-03-20 16:07:40.687 ServerApp] http://10.0.0.179:8888/lab?token=06452daa538f08d3ef351ca243735d0a97ac1cca2a95f999
[I 2023-03-20 16:07:40.687 ServerApp] or http://127.0.0.1:8888/lab?token=06452daa538f08d3ef351ca243735d0a97ac1cca2a95f999
[I 2023-03-20 16:07:40.687 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 2023-03-20 16:07:40.717 ServerApp] No web browser found: could not locate runnable browser.
[C 2023-03-20 16:07:40.719 ServerApp]
To access the server, open this file in a browser:
file:///home/petalinux/.local/share/jupyter/runtime/jpserver-879-open.html
Or copy and paste one of these URLs:
http://10.0.0.179:8888/lab?token=06452daa538f08d3ef351ca243735d0a97ac1cca2a95f999
or http://127.0.0.1:8888/lab?token=06452daa538f08d3ef351ca243735d0a97ac1cca2a95f999On the PC side, the jupyter notebook can be accesed by copying the link that contains the token. For example, in my case this was:
http://10.0.0.179:8888/?token=6dae07dd4168bf26850904cadd43526fe146fcc59de4b384
http://10.0.0.179:8888/?token=6dae07dd4168bf26850904cadd43526fe146fcc59de4b384We will implement a live video in a jupyter notebook. The recipe was taken from the following reference:
In the "Notebook" section of the "Launcher", click on the "Python3 (ipykernel)" button to create a new jupyter notebook.
Paste the following code snippets (taken directly from the above reference):
import matplotlib.pyplot as plt
import cv2
import numpy as np
from IPython.display import display, Image
import ipywidgets as widgets
import threading
# Stop button
# ================
stopButton = widgets.ToggleButton(
value=False,
description='Stop',
disabled=False,
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # (FontAwesome names without the `fa-` prefix)
)
# Display function
# ================
def view(button):
cap = cv2.VideoCapture(0)
display_handle=display(None, display_id=True)
i = 0
while True:
_, frame = cap.read()
frame = cv2.flip(frame, 1) # if your camera reverses your image
_, frame = cv2.imencode('.jpeg', frame)
display_handle.update(Image(data=frame.tobytes()))
if stopButton.value==True:
cap.release()
display_handle.update(None)
# Run
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()Save the jupyter notebook with a meaningful name (ie. usbcam_live_view.ipynb), then click the "Run" icon on each of the two code cells.
You will see a live video feed in the jupyter notebook, as shown below:
Click the "Stop" widget to stop the live feed.
Building u96v2_sbc_dualcamThe next design we will build is the "u96v2_sbc_ dualcam" design.
If not done so already, call your 2022.2 setup scripts:
$ source /tools/Xilinx/Vitis/2022.2/settings64.sh
$ source /tools/Xilinx/petalinux-v2022.2-final/settings.shNext, launch the petalinux build script as follows:
$ cd ~/Avnet_2022_2/petalinux
$ scripts/make_u96v2_sbc_dualcam.shThis script will perform the following:
- Create and Build the Vivado project
~/Avnet_2022_2/hdl/projects/u96v2_sbc_dualcam_2022_2 - Create and Build the Petalinux project
~/Avnet_2022_2/petalinux/projects/u96v2_sbc_dualcam_2022_2 - Create a final SD card image
~/Avnet_2022_2/petalinux/projects/u96v2_sbc_dualcam_2022_2/images/linux/rootfs.wic
In order to execute the base design, we first need to program the SD card image to a micro-SD card (of size 16GB or greater).
~/Avnet_2022_2/petalinux/projects/u96v2_sbc_dualcam_2022_2/images/linux/rootfs.wic
To do this, we use Balena Etcher, which is available for most operating systems.
Once programmed, insert the micro-SD card into the Ultra96-V2, and connect up the platform as shown below.
Make sure the DualCam mezzanine has the jumpers configured as follows:
- J1 => IAS0
- J2 => IAS1
- J3 => 96Boards High-Speed Connector
- J4 => 96Boards Low-Speed Connector
- J5 => MIPI DSI
- P1 – 1-2 => SENSOR1_GPIO1 is +2V8_AF
- P2 – 2-3 => SENSOR2_GPIO1 is 96B_SP3
- P3 – present => SENSOR1_GPIO0/FLASH is GND
- P4 – present => SENSOR2_GPIO0/FLASH is GND
- P5 – 2-3 => SENSOR1_GPIO3 is GND
- P6 – 2-3 => SENSOR2_GPIO3 is SENS2_ADDR
- P7 – 1-2 => PWR_SEL is 5V
- P8-P9 – P8.2-P8.3 => VDD_SEL is 1.2V
- P10 – 1-2 => VAA_SEL is 2.7V (3-2 => 2.8V)
- P11 – 2-3 => CLK_SEL is on-board OSC(48 MHz)
Press the power push-button to boot the board, and login as "root".
Next we can perform a quick sanity check with the dual MIPI sensor module.
We can query the presence of the MIPI sensors with the following commands:
root@u96v2-sbc-dualcam-2022-2:~# ls /dev/media*
/dev/media0 /dev/media1
root@u96v2-sbc-dualcam-2022-2:~# ls /dev/video*
/dev/video0 /dev/video1 /dev/video2
root@u96v2-sbc-dualcam-2022-2:~#Note that the USB camera has still been enumerated as media node /dev/media0 and video nodes /dev/video0 and /dev/video1. The dual MIPI capture pipeline has been enumerated at the next available nodes, which are /dev/media1 and /dev/video2. If the USB camera were not plugged in, the MIPI capture pipeline would have been enumerated as /dev/media0 and /dev/video0.
root@u96v2-sbc-dualcam-2022-2:~# media-ctl -p -d /dev/media1
Media controller API version 5.15.36
Media device information
------------------------
driver xilinx-video
model Xilinx Video Composite Device
serial
bus info
hw revision 0x0
driver version 5.15.36
Device topology
...
root@u96v2-sbc-dualcam-2022-2:~# v4l2-ctl -D -d /dev/video2
Driver Info:
Driver name : xilinx-vipp
Card type : vcap_CAPTURE_PIPELINE_v_proc_ss
Bus info : platform:vcap_CAPTURE_PIPELINE_
Driver version : 5.15.36
Capabilities : 0x84201000
Video Capture Multiplanar
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04201000
Video Capture Multiplanar
Streaming
Extended Pix Format
...By default, the petalinux project is configured for the AR0144 sensors, which are the sensor modules that are shipped with the DualCam mezzanine.
root@u96v2-sbc-dualcam-2022-2:~# media-ctl -p -d /dev/media1 | grep ar
<- "4-003c.ar0144.0":0 [ENABLED]
<- "4-003c.ar0144.1":0 [ENABLED]
- entity 21: 4-003c.ar0144.0 (1 pad, 1 link)
- entity 25: 4-003c.ar0144.1 (1 pad, 1 link)The petalinux project also provides some python scripts to configure the MIPI capture pipeline and stream the video via GStreamer and OpenCV. Launch the "avnet_dualcam_passthrough.py" python script from the serial console as follows:
root@u96v2-sbc-dualcam-2022-2:~# cd ~/avnet_dualcam_python_examples/
root@u96v2-sbc-dualcam-2022-2:~/avnet_dualcam_python_examples# export DISPLAY=:0.0
root@u96v2-sbc-dualcam-2022-2:~/avnet_dualcam_python_examples# python3 avnet_dualcam_passthrough.py --sensor ar0144 --mode dual
[INFO] input resolution = 640 X 480
[INFO] fps overlay = False
[INFO] brightness = 256
[INFO] Initializing the capture pipeline ...
[DualCam] Looking for devices corresponding to AP1302
dev_video = /dev/video2
dev_media = /dev/media1
sensor_type = ar0144
ap1302_i2c = 4-003c
ap1302_dev = ap1302.4-003c
ap1302_sensor = 4-003c.ar0144
[DualCam] Looking for base address for MIPI capture pipeline
mipi_desc = b0000000.mipi_csi2_rx_subsystem
csc_desc = b0020000.v_proc_ss
scaler_desc = b0040000.v_proc_ss
[DualCam] hostname = u96v2
[DualCam] Detected 96Boards dualcam (sensors placed right-left on board)
[DualCam] Initializing AP1302 for dual sensors
media-ctl -d /dev/media1 -l '"4-003c.ar0144.0":0 -> "ap1302.4-003c":0[1]'
media-ctl -d /dev/media1 -l '"4-003c.ar0144.1":0 -> "ap1302.4-003c":1[1]'
[DualCam] Initializing capture pipeline for ar0144 dual 640 480
media-ctl -d /dev/media1 -V "'ap1302.4-003c':2 [fmt:UYVY8_1X16/2560x800 field:none]"
media-ctl -d /dev/media1 -V "'b0000000.mipi_csi2_rx_subsystem':0 [fmt:UYVY8_1X16/2560x800 field:none]"
media-ctl -d /dev/media1 -V "'b0000000.mipi_csi2_rx_subsystem':1 [fmt:UYVY8_1X16/2560x800 field:none]"
media-ctl -d /dev/media1 -V "'b0020000.v_proc_ss':0 [fmt:UYVY8_1X16/2560x800 field:none]"
media-ctl -d /dev/media1 -V "'b0020000.v_proc_ss':1 [fmt:RBG24/2560x800 field:none]"
media-ctl -d /dev/media1 -V "'b0040000.v_proc_ss':0 [fmt:RBG24/2560x800 field:none]"
media-ctl -d /dev/media1 -V "'b0040000.v_proc_ss':1 [fmt:RBG24/1280x480 field:none]"
[DualCam] Disabling Auto White Balance
v4l2-ctl --set-ctrl white_balance_auto_preset=0 -d /dev/video2
[DualCam] Opening cv2.VideoCapture for 1280 480
GStreamer pipeline = v4l2src device=/dev/video2 io-mode="dmabuf" ! video/x-raw, width=1280, height=480, format=BGR, framerate=60/1 ! appsink
[ WARN:0] global /usr/src/debug/opencv/4.5.2-r0/git/modules/videoio/src/cap_gstreamer.cpp (1081) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
[DualCam] Setting brightness to 256
v4l2-ctl --set-ctrl brightness=256 -d /dev/video2This will display the stereo video from the dual MIPI capture pipeline in a window on the DisplayPort output.
The "media-ctl" and "v4l2-ctl" commands configure the following MIPI capture pipeline.
The "dual" mode captures the stereo images as shown in the following diagram.
The primary sensor (1) corresponds to the right image, and the secondary sensor (2) corresponds to the left image.
You can also stream video from the primary (right) sensor as follows:
root@u96v2-sbc-dualcam-2022-2:~/avnet_dualcam_python_examples# python3 avnet_dualcam_passthrough.py --sensor ar0144 --mode primaryYou can also stream video from the secondary (left) sensor as follows:
root@u96v2-sbc-dualcam-2022-2:~/avnet_dualcam_python_examples# python3 avnet_dualcam_passthrough.py --sensor ar0144 --mode secondaryOnce agsin, we will add jupyter notebook functionality, which is provided by "packagegroup-petalinux-jupyter".
Edit the following file:~/Avnet_2022_2/petalinux/u96v2_sbc_dualcam_2022_2/roject-spec/meta-avnet/recipes-core/images/petalinux-image-minimal.bbappend
Add "packagegroup-petalinux-jupyter" to the "IMAGE_INSTALL:append:u96v2-sbc" entry:
IMAGE_INSTALL:append:u96v2-sbc = "\
...
packagegroup-petalinux-jupyter \
"Next, rebuild the petalinux project:
$ cd ~/Avnet_2022_2/petalinux/u96v2_sbc_base_2022_2
$ petalinux-buildOnce complete, reprogram the micro-SD card, and reboot the Ultra96-V2.
Once booted, launch the jupyter-lab server in the serial console, specifying the IP address, as follows:
root@u96v2-sbc-dualcam-2022-2:$ jupyter-lab --allow-root --ip 10.0.0.179 &
[1] 879
[I 2023-03-20 16:07:40.475 ServerApp] jupyterlab | extension was successfully linked.
[I 2023-03-20 16:07:40.548 ServerApp] Writing Jupyter server cookie secret to /home/petalinux/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2023-03-20 16:07:40.649 LabApp] JupyterLab extension loaded from /usr/lib/python3.9/site-packages/jupyterlab
[I 2023-03-20 16:07:40.649 LabApp] JupyterLab application directory is /usr/share/jupyter/lab
[I 2023-03-20 16:07:40.681 ServerApp] jupyterlab | extension was successfully loaded.
[I 2023-03-20 16:07:40.686 ServerApp] Serving notebooks from local directory: /home/root
[I 2023-03-20 16:07:40.686 ServerApp] Jupyter Server 1.13.5 is running at:
[I 2023-03-20 16:07:40.687 ServerApp] http://10.0.0.179:8888/lab?token=06452daa538f08d3ef351ca243735d0a97ac1cca2a95f999
[I 2023-03-20 16:07:40.687 ServerApp] or http://127.0.0.1:8888/lab?token=06452daa538f08d3ef351ca243735d0a97ac1cca2a95f999
[I 2023-03-20 16:07:40.687 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 2023-03-20 16:07:40.717 ServerApp] No web browser found: could not locate runnable browser.
[C 2023-03-20 16:07:40.719 ServerApp]
To access the server, open this file in a browser:
file:///home/petalinux/.local/share/jupyter/runtime/jpserver-879-open.html
Or copy and paste one of these URLs:
http://10.0.0.179:8888/lab?token=06452daa538f08d3ef351ca243735d0a97ac1cca2a95f999
or http://127.0.0.1:8888/lab?token=06452daa538f08d3ef351ca243735d0a97ac1cca2a95f999On the PC side, the jupyter notebook can be accesed by copying the link that contains the token. For example, in my case this was:
http://10.0.0.179:8888/?token=6dae07dd4168bf26850904cadd43526fe146fcc59de4b384We will once again implement a live video feed in a jupyter notebook, but this time with the dual MIPI capture pipeline.
In Jupyter-Lab, navigate to the "avnet_dualcam_python_examples" directory.
In the "Notebook" section of the "Launcher", click on the "Python3 (ipykernel)" button to create a new jupyter notebook.
Paste the following code snippets:
import matplotlib.pyplot as plt
import cv2
import numpy as np
from IPython.display import display, Image
import ipywidgets as widgets
import threading
# Load dualcam helper script
import sys
import os
sys.path.append(os.path.abspath('../'))
sys.path.append(os.path.abspath('./'))
from avnet_dualcam.dualcam import DualCam
# Stop button
# ================
stopButton = widgets.ToggleButton(
value=False,
description='Stop',
disabled=False,
button_style='danger', # 'success', 'info', 'warning', 'danger' or ''
tooltip='Description',
icon='square' # (FontAwesome names without the `fa-` prefix)
)
# Display function
# ================
def view(button):
#cap = cv2.VideoCapture(0)
dualcam = DualCam('ar0144','dual',640,480)
display_handle=display(None, display_id=True)
i = 0
while True:
#_, frame = cap.read()
#frame = cv2.flip(frame, 1) # if your camera reverses your image
left,right = dualcam.capture_dual()
frame = cv2.hconcat([left,right])
_, frame = cv2.imencode('.jpeg', frame)
display_handle.update(Image(data=frame.tobytes()))
if stopButton.value==True:
cap.release()
display_handle.update(None)
# Run
# ================
display(stopButton)
thread = threading.Thread(target=view, args=(stopButton,))
thread.start()Save the jupyter notebook with a meaningful name (ie. usbcam_live_view.ipynb), then click the "Run" icon on each of the two code cells.
You will see a live video feed in the jupyter notebook, as shown below:
Click the "Stop" widget to stop the live feed.
ConclusionI hope this tutorial will help you to get your custom AI applications up and running quickly on the Ultra96-V2.
If you would like to have the pre-built petalinux BSPs or SDcard images for these designs, please let me know in the comments below.
Revision History2023/03/27
Add description of being part of a 4 part series.
2023/03/20
Preliminary Version
Comments
Please log in or sign up to comment.