Benchmarking the Kria KV260 AI Vision Starter Kit

Introduction

In this project, I'll be walking through the various methods of how benchmark the performance of the new Kria SoM development kit, the Kria KV260 Vision AI Starter Kit. The embedded Linux image released by Xilinx for the board contains commands that allow users to measure benchmarks of the SoM such as system power draw and the performance of the accelerated computer vision applications in terms of end-to-end latency and how the different network models impact the performance of the accelerated applications. This project will focus on the Smart Camera accelerated application specifically that is one of the free accelerated applications now available in the Xilinx Embedded Applications Store. See my last tutorial on Getting Started with the Kria KV260 if you're not already familiar with how to install and run accelerated applications on the Kria.

This tutorial also covers how to switch between the video source and output options of the Smart Camera accelerated application as well as the various network models.

Measuring the Power the Kria SoM Draws

A fun bonus for end users of the KV260 board is that Xilinx added dedicated power measurement capabilities that were originally intended to facilitate development, but remained in the final embedded Linux image for the development kit.

This power measurement functionality is one of the many things built into Xilinx's new platform management utility, xmutil, in embedded Linux. Simply pass the argument platformstats with the -p flag to see a range of statistics for the SoM delimited by headers defining what each section is.

root@xilinx-k26-starterkit-2020_2:~# xmutil platformstats -p

Scroll down to the Power Utilization section to see the power, current, and voltage levels drawn by the system. It's also important to note that these power measurements are also strongly influenced by the environment the device is operating in. For example, changing the temperature in the room will cause the power reading to change.

Benchmarking the Smart Camera Accelerated Application

Focusing on the Smart Camera application, the following steps walk through how to measure each available benchmark for it from the Linux console on the Kria KV260.

1 - FPS Throughput

When the Smart Camera application is ran using the smartcam command passing the -R flag, it will output the frames per second stats to the console.

root@xilinx-k26-starterkit-2020_2:~# sudo smartcam --usb 0 -W 1920 -H 1080 -r 30 --target dp --aitask facedetect -R

This allows users to measure the throughput in frames per second of the Smart Camera accelerated application (FPS). I did find that for the current embedded Linux image for the KV260 this feature only works when the output is targeted to a monitor and is not available when streaming the video signal over the wired ethernet.

After observing the FPS throughput of the application in the console, kill the application using control+C from the Linux command line.

2 - Power Draw

To measure the power drawn by the Smart Camera app run it again, but in the background this time. To run the Smart Camera app, append the & symbol to the end of the command line:

root@xilinx-k26-starterkit-2020_2:~# sudo smartcam --usb 0 -W 1920 -H 1080 -r 30 --target dp --aitask facedetect &

Hit the enter key to regain control of the console.

Then, as shown in the previous section, run the xmutil command with argument platformstats and observe the Power Utilization section in the output in the console.

root@xilinx-k26-starterkit-2020_2:~# xmutil platformstats -p

And kill the application since it's still running in the background:

root@xilinx-k26-starterkit-2020_2:~# sudo killall -9 smartcam

3 - End-to-End Latency

Accelerated applications are most commonly benchmarked by the measurement of time that it takes a buffer of data to travel from one point in the pipeline to another. The latency from point A to point B in the pipeline is known as interlatency. To measure this, there are interlatency tracer applications such as GstShark can be used.

The embedded Linux image for the KV260 does not already have GstShark installed, so connect the KV260 to the internet and use the dnf command to see what GstShark packages are available to install.

root@xilinx-k26-starterkit-2020_2:~# dnf search GstShark

Install the the libgstshark0 package:

root@xilinx-k26-starterkit-2020_2:~# sudo dnf install libgstshark0.aarch64

Say yes if prompted during the installation and wait for the console to output "Complete!".

Run the Smart Camera app again but using the GstShark interlatency tracers this time. This is done by passing GST_DEBUG="GST_TRACER:7" GST_TRACERS=interlatency before the command to run the Smart Camera application. The application also needs to be run in the background as we did to measure the power since the log from GstShark needs to be opened while it is running. Thus, also append & to the end of the command.

root@xilinx-k26-starterkit-2020_2:~# sudo GST_DEBUG="GST_TRACER:7" GST_TRACERS=interlatency smartcam --usb 0 -W 1920 -H 1080 -r 30 --target dp --aitask facedetect >& aa1-facedetect-log.txt &

Again, press enter to regain control of the console. Then use grep to parse through and return all of the interlatency values of all the plugins in the Smart Camera application GStreamer pipeline from the GstShark log file.

root@xilinx-k26-starterkit-2020_2:~# sudo grep -rn interlatency aa1-facedetect-log.txt | tail -n 20 | cut -d'=' -f2-4

The total end-to-end latency of the Smart Camera application is measured from the first source plugin, videosrc_src which is the video feed from the camera itself, to the sink plugin, kmssink0_sink which is the video sink that is rendering the images in the direct rendering manager (DRM) of the Linux kernel.

In my particular setup, using a Logitech USB web camera, the end-to-end latency of the Smart Camera measures at 0.61126 seconds or 611.26 milliseconds. Keep in mind that this reading will change upon any firmware/system updates.

Again, after completing the measurement, kill the application since it's still running in the background:

root@xilinx-k26-starterkit-2020_2:~# sudo killall -9 smartcam

How to Change the Network Model Used by the Smart Camera Application

There are three network models available to use in the Smart Camera application: face detect (facedetect), single shot detector (ssd), and refinement neural network for object detection (refinedet). To switch between network models, use the aitask flag with the desired network model facedetect, ssd, or refinedet when running the Smart Camera application.

Running the Smart Camera application with the refinement neural network for object detection network model:

root@xilinx-k26-starterkit-2020_2:~# sudo smartcam --usb 0 -W 1920 -H 1080 -r 30 --target dp --aitask refinedet

Running the Smart Camera application with the single shot detector network model:

root@xilinx-k26-starterkit-2020_2:~# sudo smartcam --usb 0 -W 1920 -H 1080 -r 30 --target dp --aitask ssd

Running the Smart Camera application with the face detection network model:

root@xilinx-k26-starterkit-2020_2:~# sudo smartcam --usb 0 -W 1920 -H 1080 -r 30 --target dp --aitask facedetect

How to the Switch Video Source of the Smart Camera Application

There are two interface options available for the Smart Camera application: USB and MIPI. The basic accessory pack Xilinx sells for the KV260 Starter Kit comes with a IAS AR1335 camera model that uses an MIPI interface.

The benchmarks we walked through in the previous steps was all done using a Logitech USB web camera, so to gain a more intuitive understating of these benchmarks of the Smart Camera application, run them all again using an MIPI interface camera.

The first flag passed to the smartcam command should be the video source specifier which is either mipi or usb. In the case of USB, the device node number also needs to be passed since the Kria KV260 baseboard contains more than one USB port.

Running the Smart Camera application with a MIPI interface camera:

root@xilinx-k26-starterkit-2020_2:~# sudo smartcam --mipi -W 1920 -H 1080 -r 30 --target dp --aitask facedetect

Running the Smart Camera application with a USB interface camera:

root@xilinx-k26-starterkit-2020_2:~# sudo smartcam --usb 0 -W 1920 -H 1080 -r 30 --target dp --aitask facedetect

How to Switch Video Stream Output Target of the Smart Camera Application

The final processed video stream with the selected network model can be output over the KV260's HDMI port or a wired Ethernet connection. This is specified by the target flag on the smartcam command. Passing dp will target the HDMI port while rtsp targets the Ethernet.

Running the Smart Camera application outputting to an HDMI monitor:

root@xilinx-k26-starterkit-2020_2:~# sudo smartcam --usb 0 -W 1920 -H 1080 -r 30 --target dp --aitask facedetect

Running the Smart Camera application outputting over a wired Ethernet connection:

root@xilinx-k26-starterkit-2020_2:~# sudo smartcam --usb 0 -W 1920 -H 1080 -r 30 --target rtsp --aitask facedetect

You'll need to use an application on the host PC such as VLC player or ffplay to view the RTSP feed on its corresponding IP, port, and channel as specified via the console by the smartcam command.