Kria KV260 FPGA ISP - Getting Started

Image Signal Processors and Infinite-ISP

An Image Signal Processor (ISP) interfaces with the image sensor, receives RAW pixels from it and transforms them into an image that matches the human perception of the captured scene. The ISP is an essential and fundamental hardware processor in every camera and vision system. However, most hardware ISP designs are proprietary in nature and the implementation details are not publicly available. Moreover, ISP makers treat the ISP processing pipeline algorithms as a unique secret sauce to stay market competitive. In open source, mostly software ISP algorithm pipelines are available for experimentation and exploration, which lack in performance, image quality and practical utility when compared to an actual hardware camera or an imaging system.

Figure: Camera Imaging Algorithm Pipeline in the ISP, Credits Karaimer and Brown, ECCV2016

Figure: Top-level Block Diagram for an Imaging SoC, Credits Lluis Gomez

Enter Infinite-ISP, a game-changer in the imaging and vision community. Infinite-ISP makes all components of the hardware Image Signal Processor development cycle open source and documents the entire development journey, taking a floating-point ISP pipeline to its hardware implementation. Revealing the inner workings of the once proprietary hardware ISP, builds collective knowledge, accelerates development and encourages contribution from the entire imaging and vision community, be it professionals, researchers or open source hardware enthusiasts.

The Infinite-ISP Suite

Infinite-ISP is an open-source hardware image signal processor development suite, complete with an Algorithm Development Floating Point Model, a Fixed-Point Reference Model (Golden Model), an RTL ISP, FPGA / ASIC implementation and ISP Tuning Tool application. The open source ISP design package offers competitive image quality and performance of a baseline modern image signal processor.

Using the Infinite-ISP development suite, we have developed the Infinite-ISP hardware image signal processor in Verilog (RTL ISP). The RTL ISP is FPGA proven with the following specs:

• 10-bit 4 Megapixels Image Signal Processor for Bayer image sensors
• Max. Frame Size: 2592 x 1536
• Max. Frame Rate: 30 fps
• Pixel Throughput: Up to 125 Mpix/s (can increase frame size at cost of fps)
• Selectable ITU-R compliant YCbCr outputs:
• BT.601
• BT.709
• Selectable RGB, YCbCr444, YCbCr422 outputs
• 2A Statistics Engine (AWB, AE)
• Multi-level Noise Reduction (Bayer, Luma)
• Configurable processing pipeline with ISP Tuning Tools support

For the FPGA implementation of RTL ISP, we chose AMD Xilinx Kria KV260 Vision AI Starter kit featuring an AMD Xilinx Zynq UltraScale+ XCK26 FPGA device. The kit is very suited to rapid prototyping and development of Imaging System and Vision Solutions, providing all the necessary interfaces for image sensors and display at competitive pricing.

We publish a series of Tutorials to help you understand the Infinite-ISP development suite, familiarize with its various components and the features offered in each of them, and work with them to improve the RTL ISP or tune it for your application. Each component of the package offers unique value and can be used independently. The hardware ISP is available as both standalone IP and as part of a complete sensor-to-display FPGA Imaging System.

Through the tutorial series, we broadly enable the imaging and vision community to better understand camera pipeline hardware and practical image signal processors, design sophisticated ISP algorithms dealing with uncompressed image sensor data and improve upon the baseline ISP that is more adaptive and robust in targeted applications and market segments (Surveillance, Automotive, Medical etc).

What this Tutorial is about

This tutorial will walk you through the deployment of Infinite-ISP FPGA binaries on the AMD Xilinx Kria KV260 Vision AI Starter Kit, converting it into a usable camera system with fully accessible open source hardware image signal processor. You will be able to observe the captured video stream from an image sensor as output of the ISP on an HDMI screen in real time.

Getting Started with Infinite-ISP on AMD Xilinx Kria KV260 AI Starter Kit

Hardware Setup:

To run Infinite-ISP on the Kria KV260 Kit, the following items are required:

• Kria KV260 Vision AI Starter Kit
• One of the following 3 camera modules:
• Raspberry Pi Camera Module v2.0
• + 15-pin FFC cable 0.5mm pitch Type B (reversed)
• Raspberry Pi Camera Module v1.3
• + 15-pin FFC cable 0.5mm pitch Type B (reversed)
• AR1335 IAS Module
• Ethernet Cable
• USB A to USB micro C data cable
• Power adaptor
• HDMI Cable
• Display monitor with HDMI port

The hardware connectivity for the Kira KV260 Vision AI Starter Kit is shown as follows:

Bringing Up the Kria KV260 kit with Infinite-ISP Binaries:

Infinite-ISP currently supports three different image sensors, with separate binaries for each of them. These binaries can be found on GitHub and are all publicly accessible. The supported sensors are:

• Onsemi AR1335
• Omnivision OV5647
• Sony IMX219

The AR1335 is available as an IAS sensor module and must be plugged into the IAS1 port as shown in the figure, while the OV5647 or the IMX219 sensors must be plugged into the 15-pin RPi connector as shown in the figure.

For the sake of simplicity, we will move ahead and discuss in detail the Infinite-ISP Kria KV260 Binary for Onsemi AR1335 image sensor. The process of using the FPGA binaries is the same for other image sensors and can be replicated the same.

Selecting the Infinite-ISP Binary file for Kria KV260 kit

Download the binary you want to flash the board with. For AR1335 image sensor, select the ‘Infinite-ISP_v1.4-AR1335.bin’ from the repo for flashing on the FPGA kit (or check for latest release with AR1335 image sensor):

Flashing the Infinite-ISP Binary file on Kria KV260 kit

You can skip to the next section if you are already familiar with the detailed process of flashing a binary file on the Kria KV260 kit using the Boot Image Recovery Tool method.

Now that you have selected the right binary file for the Kria KV260 kit, you need to properly flash the kit with the binary file and reset the Kria KV260 kit after flashing is complete to get the design running.

The process is well-documented in AMD Xilinx documentation for the Kria KV260 kit here, but for the sake of user convenience we include the steps in detail in this section.

To flash the Kria KV260 Kit with an Infinite-ISP binary, follow the aforementioned steps:

1. Change your PC's IP address to a static IP 192.168.0.X, where X can be any number between 2 and 254 except for 111. The Netmask and the Gateway should be 255.255.255.0 and 192.168.0.1 respectively.

2. Connect the Ethernet cable with the Kria KV260 Kit and your PC.

3. Plug the power cable into the power jack of the Kria KV260 Kit.

4. Once the board is powered on, press and hold the FWUEN button with one hand, press and release the RESET button with the other hand while holding the FWUEN button, then release the FWUEN button 5-10 seconds after releasing the RESET button.

5. Open any internet browser on your PC and type 192.168.0.111 in the address bar as shown below:

This will open the kit’s boot image recovery tool as shown below from which you can flash your board with a different binary. If it doesn’t appear, then repeat step 5.

6. Select Image B in both of the options as shown below:

7. Then click the Browse button under the Recover Image section and give a path to the downloaded image of step 1.

8. Then click the Upload button. It will take a few seconds to update the Image B with the Infinite–ISP image.

9. You can now disconnect the Ethernet cable from your board and PC as it is now not needed.

10. Remove the power cable, and make sure the sensor whose binary you have flashed in the Kria KV260 Kit is attached to the kit.

• In the case of the AR1335 binary, the AR1335 sensor must be plugged into the IAS1 port.
• In the case of the OV5647 binary, make sure that the RPi camera module v1.3 is plugged into the RPi port.
• In the case of the IMX219 binary, make sure that the RPi camera module v2.0 is plugged into the RPi port.

11. Attach the USB cable with your PC and the Kria KV260 Kit.

12. Plug the power cable into the power jack of the Kria KV260 Kit. It will now load and run the Infinite-ISP binary on the Kria KV260 Kit.

Running Infinite-ISP on the Kria KV260 Kit

After flashing the Kria KV260 Kit with the relevant binary, attach the sensor associated with the binary to the Kria KV260 Kit. A power reset will load and run the Infinite-ISP binary on the Kria KV260 kit.

To access a user menu via serial interface, connect one end of the USB cable (USB micro) with the Kria KV260 Kit and the other end (USB A) to your PC.

Connect one end of the HDMI cable to the Kria KV260 Kit and the other end to a monitor/display. Insert the SD card into the SD card slot of the Kria KV260 Kit.

Plug in the power cable. The Infinite-ISP binary will start running on the Kria KV260 Kit.

The details of the hardware connectivity are shown in the following figure:

The output of the Infinite-ISP will be displayed on the HDMI monitor.

To interact with the Infinite-ISP in real-time, a configuration menu over serial interface is also provided through which users can tune the Infinite-ISP, change the image sensor’s configuration such as exposure and gain settings in real-time and also capture RAW and processed frames in the SD card. This is demonstrated in the next section.

Kria ISP Reference Design - User Menu

To access the Kria ISP user menu, download Putty on your PC. Select Serial connection and set the baud rate to 115200. Select the appropriate port and start the serial session. The steps for setting up putty are explained below:

Pre-requisites:

In order to access the user menu over serial interface, the serial port relevant to Kria KV260 kit needs to be accessible on your PC and you need to have a serial interface client (a popular choice is PuTTY).

• Download PuTTY for Windows from here and for Ubuntu from here.
• For Windows install FTDI drivers from here.

After the USB cable from Kria KV260 is connected to your PC, the COM port should be visible and the serial interface will be accessible to the PuTTY client.

Baud Rate: set to 115200

Navigating the Kria ISP User Menu:

Connect via the serial interface to the Kria KV260 kit through the relevant COM port. The user menu will be available at the start in which the interrupt counts from different blocks of the Imaging System implemented on the Kria Kv260, receiving the image’s height, width, and type (di) will appear as shown below:

Press any key and the Infinite-ISP Configuration Menu will appear as shown below:

The Infinite-ISP Configuration Menu contains three options. You can change sensor configurations such as exposure duration and analog/digital gain settings, start burst capture of consecutive frames, change ISP parameters, and even adjust the focus of the sensor module. These are explained individually in the coming sections.

Change Sensor Configuration:

Upon selecting option 1 of the “Infinite-ISP Configuration Menu”, you will be asked to change either the sensor’s exposure duration or the sensor’s analog/digital gains. The “Change Sensor Configuration” menu is shown below.

Upon selecting option 1 (Change Sensor Exposure) of the “Change Sensor Configuration” menu, you will be shown the current “coarse_integration_time” register value and you will be asked to give exposure duration in the units of lines_of_frame (time required by sensor clock to process one line of the frame). Upon entering any value, the exposure duration of the sensor will change. This change can be observed live on the HDMI display attached to the board.

Upon selecting option 2 (Change Sensor Gain) of the “Change Sensor Configuration” menu, you will be shown the value of the current gain values and you will be asked to modify the value of any gain register. In options 1 – 4, you will be shown the current gain register value of that particular option and you will be prompted for the new value. In option 5, the default sensor gains will be applied. In option 6, the current register values of gain registers will be displayed. In option 7, you will go back to the “Change Sensor Configuration” menu. The “Change Sensor Gain” menu is shown below:

By selecting option 2 (Analog fine gain), we can modify it and it will be reflected in the “Change Sensor Gain” menu as shown below:

Upon selecting option “S” (Save sensor exposure and gain to file) of the “Change Sensor Configuration” menu, the current exposure and gain register values will be written to “ExpAndGains_<sensor_name>.txt” file in the SD card. Whenever the board is restarted, it will check for this file and if found, it will initialize the sensor with the exposure and gain register values present in that file. In case you don’t have an SD card inserted in the board at the time of restart, it will wait for a few seconds and will initialize the sensor with default exposure and gain register values.

You can select option E (Exit) to go back to the “Infinite-ISP Configuration Menu”.

Start Burst Capture:

Upon selecting option 2 (Start Burst Capture) of the “Infinite-ISP Configuration Menu”, you will be asked to enter the number of frames you want to dump.

You can also press E/e to abort the burst capture at any time. Then you will be asked to enter the number of frames you want to skip at the start of the burst capture feature. In the case of AR1335, the Frames Per Second (FPS) are 30 and if the value you entered is 300, it will wait for floor(300 / FPS) i.e. floor(300 / 30) = 10 seconds and then it will capture the scene.

As soon as you insert the value, the feature will start and will wait for the frames you entered to be skipped. During this period, it will show you the time at which the scene will be captured and the time passed since your input. This waiting period is shown below:

Once the wait period is over, it will show you the status of the burst capture feature. The ‘In memory’ flag rises once the scene is captured and the ‘In SD card’ flag rises once the captured frames are successfully written to the SD Card. It first writes the RAW frames and then the ISPout frames. Once the frames get dumped into the SD card, it will bring you back to the “Infinite-ISP Configuration Menu”.

Configure ISP Parameters:

Several ISP modules can be configured through the “Configure ISP Parameters” menu option. The ISP configuration menu looks as follows:

Enter the module number you want to configure. Entering a valid number will lead you to the module-specific configuration menu in which you will be shown the current parameter values and you will be prompted to select a parameter you want to change/configure.

After selecting a parameter for configuration, you will be directed to the parameter-specific menu in which you will be shown the current value of the parameter and will be prompted for a new value.

Once you enter the new value, you will be directed back to the module-specific configuration menu and you will see that your change has been reflected in it.

Press ‘E’ to go back to the ISP configuration menu. You can go through the other modules and play with them to observe the changes in real time on the HDMI display.

Change VCM Position:

You can change the focus of the AR1335 sensor module by changing the VCM position register value. It will give you the current register value and will ask for the new position register value. This feature requires a board modification to work properly. For further details, leave a comment and we will be glad to assist with this.

Live Demonstration

The user menu configuration options discussed above and the effects of changing each of the configurations on ISP output can be explored and checked out in our demonstration of the latest Kria FPGA ISP Binaries with AR1335 image sensor.

Conclusion

That’s all there is to Getting Started with Infinite-ISP, the open source image signal processor on Kria KV260 Vision AI Starter Kit. Camera enthusiasts and developers alike can use the provided FPGA binaries, quickly deploy a Camera System on the Kria KV260 kit, control image sensor and ISP parameters at runtime to see the effects of changes in processing, and capture sensor RAW and ISP output images on the go.

In the next tutorial we will explain how to build the Infinite-ISP FPGA project on Kria KV260 kit from source using the Vivado Toolchain.

Leave your valuable feedback and comments below. Let us know if you have any questions and suggestions related to this tutorial and Getting Started with Infinite-ISP on Kria KV260 Vision AI Starter Kit.