Introduction to Xilinx Kria AI Smart Camera Applications

Introduction

Xilinx have introduced Kria K26 System On Module (SOM) built around Zynq Ultrascale+ MPSoC. The Kria K26 SOM is mounted on Xilinx carrier card “Kria KV260 Vision AI Starter Kit” to be used only development and evaluation purpose. The Kria KV260 Vision AI starter Kit is targeting Vision AI applications. Kria SOM platform is enabled with a number of accelerated applications that can be dynamically installed and customized.

The Kria K260 Vision AI Start Kit platform [1] is made up of four pipeline stages:

1.Capture pipeline, captures video frames from a file on storage device SD card, or USB webcam, or image sensor camera into DDR memory.

2. Video processing pipeline performs video encoding/decoding, and video compression/decompression.

3. Acceleration pipeline uses programmable Deep Learning Processing Unite (DPU) Convoutional Neural Network IP to perform acceleration functions. Where for Smart Camera Applications Densebox_640_360 network model is used for face detection. For car, bicycle, person detection ssd_adas_pruned_0_95, pruned 0.95 network model is used. For pedestrian detection refinedet_pruned_0_96 pruned 0.95 network model is used.

4. Output pipeline reads video frames from DDR memory and send them to output device or file. The output could be stored as a file, or passed forward via Ethernet using RTSP protocol, or displayed on monitor.

1. Kria SOM Smart Camera Applications

The Kria SOM smart camera applications perform face detection, car, bicycle, and person detection. This tutorial shows how easily face detection can be applied to a video file on Kria K260 Vision AI starter kit on Windows machine. The input video file “Alley - 39837.mp4” used for face detection in this tutorial can be downloaded from reference [2] into your local PC drive. The input MP4 video file is transcode into H262 format with (1920x1080) resolution. The format and resolution conversion is performed using the FFmpeg [3] multimedia framework, which can be downloaded into local PC. In FFmpeg download link [3], under release header, download the FFmpeg release essentials image [4], “ffmpeg-4.4-essentials_build.7z “.

Unzipp the “ffmpeg-4.4-essentials_build.7z”, image and also place the input video MP4 file “Alley - 39837.mp4” under the ffmpeg bin folder as shown below.

• Open the DOS command on your windows PC, in the ffmpeg bin folder transcode the input MP4 video file into H262, and (1920x1080) resolution format using the following command.

ffmpeg -i “Alley - 39837.mp4” -c:v libx264 -pix_fmt nv12 -vf scale=1920:1080 -r 30 output.alley.nv12.h264

• Place the H264 transcoded output file “output.alley.nv12.h264” into FFmpeg bin folder.

2. Kria SOM Starter Kit Setup

• Dowland Kria SOM plattform petalinux image “petalinux-sdimage.wic.gz” [5] into your local PC.
• Insert SD card reader containing 16G Micro SD card into your PC card reader slot.
• Download the Balena Etcher tool [6] into your PC.
• Open Balena Etcher tool and browser to the location of the downloaded Kria SOM petalinux image. In Balena Etcher tool select the blue select image button.
• In Balena Etcher from the device drop-down menu, select the correct micro SD card to flash Xilinx Kria SOM Smart Camera Application image.
• Open the downloaded “petalinux-sdimage.wic.gz” image in Etcher and flash the image into Micro SD card as shown in the diagram.

• After flashing the Kria petalinux image “petalinux-sdimage.wic.gz” into micro SD card. Remove the micro SD card from the PC card reader.

3. Kria KV260 Vision AI Starter Kit Connection Setup

• Insert the micro SD card containing the petalinux boot image into the SD card slot (J11) on the Kira SOM Starter Kit board.
• Connect micro-USB cable B end to J4 on the Kria Starter Kit.
• Connect your local Ethernet cable to Kria Starter Kit Ethernet port.
• Connect the power supply cable to the DC Jack (J12) on the Starter Kit. Do not insert the other end to the AC – power supply plug yet.

4. Booting Kria KV260 Starter Kit from SD Card

• After connecting the USB-A end to your computer. The Starter Kit uses an FTDI USB to COM port driver, which will be detected by host PC.
• The lower numbered COM port corresponds to the UART, which should be selected for serial port communication to the Kria starter Kit.
• Install Tera Term [7].
• Open Tera Term and select correct USB serial Port for Kria SOM Starter kit as shwon below.

• From Tera Term tool menu bar at the top select Setup Serial Port, and make sure you have the correct settings as follow.

• Power ON the Kria SOM Starter Kit by connecting the Kria SOM Starter Kit power supply cable to the AC 12V 3AMP power supply. The power LEDs should illuminate and a Linux UART response should be seen on the Tera Term terminal interface.
• The Kria Starter Kit will boot from the petalinux image in the SD card inserted into Kria starter Kit. The platform default login username is “petalinux” and the user has to set a new user password of their choice.

• Verify Internet connection using “ping” command.

5. Launching the Smart Camera Accelerated Application

Xilinx provides package feeds in run-time package management RPMformat for users to dynamically load Xilinx accelerated applications. Xilinx provides package manager utility dnf, and xmutil utility.

• Update to the latest available packages using dnf update command “sudo dnfupdate”.
• The xmutil utility can be used to manage accelerated applications and obtaining Kria SOM Starter Kit platform status information. For further details on xmutil utility usage use “sudo xmutil –h” xmutil help command.
• To view the latest available application package groups run “sudo xmutil getpkgs” command.

• Install Smart Camera accelerated application package group from the above list and press “y” when prompted.

Packages are installed from the RPM feed onto the Kria SOM Linux image target root file system. A summary of installed packages and complete message are displayed when package installation is successful.

5.1. Load Smart Camera Accelerated Application

The newly installed applications and design specific files such as the Programmable Logic (PL) bitstream, device tree overlay, and PL drivers are available under “/opt/Xilinx/bin” folder.

• List existing applications available on the Kria Starter kit.

On initial boot a default configuration is loaded into the starter Kit “kv260-dp”, which enables the display port interface. The current active application is indicated by active state has a value of “1” as shown above.

• Before loading a new application, currently active application has to be unloaded using "sudo xmutil unloadapp" command.

• Then load smart camera accelerated application using “sudo xmutil loadapp kv260-smartcam” command.

The command prints the output of newly loaded drivers based on the device tree of the overlay that corresponds to the PL bitstreams. At the end of command completion a successfully load message will be printed.

• Verify “kv260-smartcam” application installation by using “sudo xmutil listapps” command.

5.2 Run Smart Camera Application

The installed applications can be run using either:

• Command line
• Bash scripts
• Juypter notebook

In this tutorial smart camera application is run using command line. For ease of use Xilinx have provided example scripts to run different applications.

• The available application scripts can be viewed under “/opt/Xilinx/bin” folder. First change to this folder.

• The application scripts can viewed with the “ls” command

The “03.file-file.sh”, script apply face detection to input video file.

• Verify that the h264 encoded video file “output.alley.nv12.h264”is available in micro SD card on Kria K260 starter kit. First change to SD card location on the board “/media/sd-mmcblk0p1”, and then list available files using “ls –ll” command.

• Source the face detection script as per below command. The “03.file-file.sh”, script reads the input video file “output.alley.nv12.h264”, and configures and perform face detection on Kria SOM, generating output video file “out.h264”.

sudo ./03.file-file.sh /media/sd-mmcblk0p1/output.alley.nv12.h264

• Verify and observe the generated output video file “out.h264”.

• Copy the face detection output video file “out.h264” into Micro SD card in Kria Starter Kit using "sudo cp" command.

• Shutdown Kria SOM Starter kit using “sudo shutdown -h -P now” command. Where “-P” is capital letter. All power indicator LEDs will go off, except the fan will be running from the 12V power supply. To power cycle remove the power cable from Kria SOM Kit.

• Remove the Micro SD card from Kria Starter Kit, and insert it into Windows PC SD Card reader.
• Copy the generated face detection video file “out.h264” into the same folder bin that contains ffmpeg.exe file as shown below.

• Display face detection output file “out.h264” using ffplay or VLC multimedia player [8]

Next Step:

Next Step is to repeat the above steps for Freeway video file [9], using “04.file-ssd-dp.sh” script.

References:

[1]: https://xilinx.github.io/kria-apps-docs/master/index.html

[2]: https://pixabay.com/videos/alley-people-walk-street-ukraine-39837/

[3]: https://www.gyan.dev/ffmpeg/builds/

[4]: https://www.gyan.dev/ffmpeg/builds/ffmpeg-release-essentials.7z

[5]:https://login.xilinx.com/app/xilinxinc_f5awsprod_1/exknv8ms950lm0Ldh0x7/sso/saml

[6]: https://www.balena.io/etcher/

[7]: https://www.xilinx.com/products/som/kria/kv260-vision-starter-kit/kv260-getting-started/getting-started.html

[8]: https://www.videolan.org/

[9]: https://pixabay.com/videos/freeway-traffic-cars-rainy-truck-8358/

Revision History

16/06/2021 - Kria AI Smart Camera Applications on Windows Machine