Published May 11, 2024 © Apache-2.0

Kria Adaptable KV26 Vision AI System On Module (SOM)

The project provides introduction to Kria KV26 SOM development tools and frameworks for developing custom applications.

BeginnerProtip1 hour55

Kria Adaptable KV26 Vision AI System On Module (SOM)

Story

1. Introduction to Intelligent Vision at the Edge

Intelligent vision is a branch of Artificial Intelligence (AI) that uses Machine Learning technology enabling electronic devices to interpret and understand visual information from digital images or videos. Machin Vision has massive impact across the industry from retail to security, healthcare, construction, automotive, manufacturing, logistics, and agriculture. The tasks of machine vision involve image recognition, object detection and tracking, image segmentation, scene understanding and many more.

The AI in computer vision market is estimated to grow from USD 17.2 Billion in 2023 to USD 45.7 Billion by 2028, growing at CAGR [1] of 21.5% during the forecast period.

Building Intelligent Vision System requires many different devices and processes, some of which are listed below.

Sensor cameras to capture raw visual data.
Pre-processing convert captured raw data into a suitable format for further processing by AI engine.
AI models are used for processing images to obtain meaningful insight into images for different applications.
Post-processing converts images back to a format suitable for display or transmission over the internet.
Conditional logics used to automate specific application use cases.
Video Codec Unite (VCU) is required to encode (compress) and decode (de-compress) video images to reduce required transmission bandwidth or memory storage as they are very scarce resource.

Building device based intelligent Vision systems from scratch is very complicated and time-consuming require the expertise of system design engineers, hardware engineers, embedded software engineers, AI and data scientist, system integration engineer and test validation engineers.

AMD released into the market Kria KV260 Vision AI start kit with on board K26 System On Module (SOM) for Vision AI ON the Edge applications. The Kria KV260 Vision AI Starter kit enables customers to bring their intelligent Vision products into market in very short space of time as shown in Figure.1, saving several months on design and development.

Figure.1: Kria K26 SOM products development timeline.

2. Introduction to Kria KV260 Vision AI Starter Kit

Kria KV260 Vision AI starter kit [2], and its software development ecosystem provide customers with out of the box solution for developing advanced vision applications products without the need for complex hardware knowledge. A simplified diagram of Kria Vision AI SOM KV26 [3, 4], which consists of two sub-systems is shown in Figure.2.

Processing Sub-system (PS) consist quad Core ARM Cortex A53 processors for applications and control operations. The PS also have dual core ARM Cortex R5 processors for time critical tasks.
Programmable Logic (PL) Sub-system consist of Field Programmable Gate Array (FPGA), for accelerating computationally intensive tasks.

Figure.2: Simplified diagram of Kria KV26 SOM

The PL sub-system of the Kria K26 SOM is the computational powerhouse which can process compute intensive tasks in a fraction of CPU time. The PL sub-system is highly configurable which makes the Kria KV26 SOM future proof and adaptable to different Vision AI applications.

The KV260 vision starter kit carrier card is ready with multiple camera support, built in HDMI, display port outputs, and multiple other connectivity options for application developer to quickly develop and evaluate their applications.

The Kria KV26 SOM is a game changer as AMD is basically have done the entire hardware design for the developer, no further hardware design is required, but of course, it is possible if the developer want to customize hardware or add their hardware design functionality, they can do so. Most of the time just application software design utilizing the pre-built accelerators is all that is required to move their project application forward.

The Kria KV26 SOM simplified diagram in Figure.1, shows Vision AI data processing pipeline with several AMD hardware components, which are briefly explained.

2.1: Image Signal Processor (ISP):

Kria K26 SOM ISP handle video input from camera sensors and perform image sensor control, noise reduction, and colour correction. These capabilities are very useful for intelligent vision AI applications.

2.2: Video Codec Unite (VCU):

Kria k26 SOM VCU [5] transcoder decodes received input compressed video stream back into its original format for further processing or playback. The VCU can also encode video streams into a compressed format for storage or transmission over internet.

The VCU is software programmable, which can be optimized for different applications requirements. The VCU can perform computationally heavy video stream encoding and decoding simultaneously at extremely low latency and low power consumption.

2.3: AI Deep Learning Processing Unite (DPU):

The kria K26 SOM DPU [6] is a programmable AI computation engine optimized for convolutional neural networks (CNN). The degree of parallelism utilized in the AI engine is a design parameter and can be selected according to the target device PL resources availability and application requirements. AMD Vitis AI development tool is used for running pre-trained AI models on the DPU AI engine. The DPU can be optimized for minimum latency, or high throughput or low power consumption. For extreme power or latency constrained applications the developers can further optimize the DPU using Vitis AI optimizer tool.

3.Kria KV26 SOM Development Tools

The uniqueness of Kria KV26 SOM is not only that, but it can be reconfigured and optimized for many Vision AI applications. It also comes with rich and mature development ecosystem. The developer of different skill set and persona (AI Developer, Embedded Developer, Software developer, Hardware Developer) can design, develop, evaluate performance of their applications on Kria KV260 Starter kit with their familiar development tools.

3.1. Vitis AI Development Tool

The AI Developer and Data scientists can use Vitis AI development tools [7, 8] to convert pre-trained AI models into a format which are optimized to DPU engine instruction set. All this is done at software level with no requirement for FPGA design.

3.2. Vitis Unified Software Platform Tool

Embedded Developer can design and customize their applications software targeting Kria KV26 SOM processing sub-system (PS) ARM Cortex A53, and R5 processors.

The Software Developer can change the hardware FPGA design in a high-level software programming like Python, C, C++, OpenCL using AMD Vitis [9, 10] and Vitis HLS development tools [11].

Vitis Integrated Development Environment (IDE) has several optimized libraries, which can be used out of the box to customize and accelerate Kria K26 data processing pipeline pre and post processing operations. If the application needs further customizations or hardware acceleration, the software developer can create FPGA hardware accelerated kernels using Vitis or Vitis HLS development tools.

3.3. Vivado Integrated Development Environment (IDE)

For full customization of KRIA data processing pipeline at hardware level, the developer can use Vivado IDE [12]. This is traditional FPGA design tool and it’s not required for Kria KV26 SOM, but this level of flexibility is available to those that have that expertise.

3.4. Petaliunx

Petalinux is an embedded Linux Software Development Kit (SDK) [13]. Petalinux is a simplified wrapper for Linux Yocto Operating System. Embedded Software developers can design, develop, prototype, their custom Kria embedded Linux applications on Kria KV260 Vision AI Starter Kit.

3.5. Ubuntu

The popular Ubuntu Operating System (OS) [14] support development on Kria KV260 Vision AI Starter Kit. The Embedded Software and AI Developer can develop Kria Vision AI applications on Ubuntu Linux operating system.

4. Kria Accelerated Applications APPS

The concept of accelerated applications is the real Kria K26 SOM value proposition. AMD and its partners have created a library of accelerated applications [15]. Each pre-built accelerated application transforms the K26 SOM to a different product, such as smart city camera, retail analytic, defect detection, Natural Language Processing (NLP) smart vision, health care platform, or Number Plate Recognition system and many more. Developer can download Kria accelerated applications from the Kria App store [3] and they can evaluate the application on Kria KV260 Vision AI starter kit in less than an hour. Figure.3, shows several Kria App store Applications.

Smart Camera AI Box Re-Identification Defect Detection NLP Smart Vision

Figure.3: Kria SOM App Store Accelerated Applications

One of the big advantages of the Kria K26 SOM is the ease and speed with which new applications can be created, either around the existing available applications or with new AI application dropped into the K26 SOM.

5. Vitis AI Model Zoo.

AMD AI model Zoo repository [16] have more than 130 pre-trained AI models for different applications. All model Zoo models are available as an open source. Many of these models are optimized for Kria K26 SOM DPU AI Engine. Model Zoo provides comprehensive benchmark information on the performance of these AI models for different DPU configurations and size. If Kria K26 SOM application developer need to upgrade or replace existing Kria SOM AI model, the developer can simply update their AI model to a new model from model zoo, they can also re-train these AI models to their specific application, or they can use their own models using Vitis AI development tools.

Figure.4: Vitis AI Model Zoo.

6. Vitis Video Analytic SDK (VVAS)

The VVAS software stack [17], as shown in figure.5, enables Kria K26 SOM vision AI developer to quickly compose video analytics data processing pipeline at software level. The VVAS framework on Kria KV26 takes input from USB/MIPI cameras, or from video files, or from video streams over Real Time Streaming Protocol (RTSP). The VVAS video analytic data processing pipeline uses Vitis AI to generate insights from image pixels for various use cases such as understanding traffic, pedestrians in smart cities, health and safety monitoring in hospitals, self-checkout, and analytics in retail, detecting component defects at a manufacturing facility.

The core VVAS SDK is built on top of the popular open-source multimedia video pipeline framework GStreamer [19]. For the Kria Video Analytic developer, VVAS provides a framework in the form of generic Infrastructure plugins, software acceleration libraries, software driver, and a simplified interface to develop their own acceleration library and to control a custom hardware accelerator. With this framework, for developer to easily integrate their custom accelerators/kernels into their Video Analytics data pipeline. As shown in Figure 5, Xilinx Run Time (XRT) API, Vitis, and Vitis AI abstracts complex interfaces, and making it easier to develop intelligent video analytics applications.

Further information and a step-by-step guid on how to create custom Vision AI applications using VVAS SDK is provided in Reference [18].

Figure.5: Vitis Video Analytic SDK (VVAS)

7. PYNQ on Kria KV26 SOM.

Kria KV260 Vision starter kit supports PYNQ framework [20]. PYNQ is open source Jupyter Notebook based on an interactive computing framework using Python programming language. Kria KV26 SOM applications Software developers, Data Scientists, and AI developers can prototype and evaluate their applications on Kria KV260 starter kit using PYNQ API on Jupyter notebook as shown in Figure.6. The key enabler component of PYNQ framework is the Overlays. The Overlays are Hardware FPGA IPs bitstreams developed in Vivado IDE or Vitis HLS ported to PYNQ framework. The Overlays are available as source files or bitstream and developer can also generate their own overlays. The composable overlay allows developers to compose their application data processing pipeline by connecting different IPs (available as overlays) via a switching mechanism (AXI4 Stream) using PYNQ Python Jupyter Notebook. With PYNQ framework It is also possible to reconfigure the data processing pipeline dynamically at run time.

Figure.6: PYNQ framework for Kria KV260 Vision AI Starter Kit.

8. Summary and Conclusions

Vision AI applications developer of different skill sets can design, prototype, and evaluate their application on Kria KV260 Vision AI Starter Kit in very short time. Kria K26 SOM unique reconfigurability, adaptability, low end to end processing latency, low power consumption and custom data processing pipeline acceleration makes it ideal platform for many Vision AI applications. Kria application developers can further optimize their applications to meet strict power consumption, and time critical latency requirements. Kria application developer can leverage Kira KV26 SOM development ecosystem to bring their Vision AI edge product to market in a very short time.