Titanium Ti180 FPGA ISP - Getting Started

Image Signal Processors and Infinite-ISP

An Image Signal Processor (ISP) interfaces with the image sensor, receives RAW pixels from it and transforms them into an image that matches the human perception of the captured scene. The ISP is an essential and fundamental hardware processor in every camera and vision system. However, most hardware ISP designs are proprietary in nature and the implementation details are not publicly available. Moreover, ISP makers treat the ISP processing pipeline algorithms as a unique secret sauce to stay market competitive. In open source, mostly software ISP algorithm pipelines are available for experimentation and exploration, which lack in performance, image quality and practical utility when compared to an actual hardware camera or an imaging system.

Enter Infinite-ISP, a game-changer in the imaging and vision community. Infinite-ISP makes all components of the hardware Image Signal Processor development cycle open source and documents the entire development journey, taking a floating-point ISP pipeline to its hardware implementation. Revealing the inner workings of the once proprietary hardware ISP, builds collective knowledge, accelerates development and encourages contribution from the entire imaging and vision community, be it professionals, researchers or open source hardware enthusiasts.

The Infinite-ISP Suite

Infinite-ISP is an open-source hardware image signal processor development suite, complete with an Algorithm Development Floating Point Model, a Fixed-Point Reference Model (Golden Model), an RTL ISP, FPGA, and ISP Tuning Tool application. The open source ISP design package offers competitive image quality and performance of a baseline modern image signal processor.

Using the Infinite-ISP development suite, we have developed the Infinite-ISP hardware image signal processor in Verilog (RTL ISP). The RTL ISP is FPGA proven with the following specs:

• 10-bit 2 Megapixels Image Signal Processor for Bayer image sensors
• Max. Frame Size: 1952 x 1112
• Max. Frame Rate: 20fps
• Pixel Throughput: Up to 100 Mpix/s
• Selectable RGB, YCbCr444, YCbCr422 outputs
• Multi-level Noise Reduction (Bayer, Luma)

For the FPGA implementation of RTL ISP, we chose the Efinix Titanium Ti180 J484 Development kit. The kit is very suited to rapid prototyping and development of Imaging System and Vision Solutions, providing all the necessary interfaces for image sensors and display at competitive pricing.

We publish a series of Tutorials to help you understand the Infinite-ISP development suite, familiarize with its various components and the features offered in each of them, and work with them to improve the RTL ISP or tune it for your application. Each component of the package offers unique value and can be used independently. The hardware ISP is available as both standalone IP and as part of a complete sensor-to-display FPGA Imaging System.

Through the tutorial series, we broadly enable the imaging and vision community to better understand camera pipeline hardware and practical image signal processors, design sophisticated ISP algorithms dealing with uncompressed image sensor data and improve upon the baseline ISP that is more adaptive and robust in targeted applications and market segments (Surveillance, Automotive, Medical etc).

What this Tutorial is about

This tutorial will walk you through the deployment of Infinite-ISP FPGA binaries on the Efinix Titanium Ti180 J484 Development kit, converting it into a usable camera system with fully accessible open source hardware image signal processor. You will be able to observe the captured video stream from an image sensor as output of the ISP on an HDMI screen in real time.

Hardware Setup:

To run Infinite-ISP on the Titanium Ti180 J484, the following items are required:

• Titanium Ti180 J484 Development kit
• Raspberry Pi Camera Module v2.0
• + 15-pin FFC cable 0.5mm pitch Type B (reversed)
• USB A to USB type C data cable
• Power adaptor
• HDMI Cable
• Display monitor with HDMI port

The hardware connectivity for the Titanium Ti180 J484 Development kit is shown as follows:

Bringing Up the Titanium Development kit with Infinite-ISP Binaries:

Infinite-ISP currently supports Sony IMX219 image sensor, with their binaries. These binaries can be found on GitHub and are all publicly accessible. The IMX219 sensors must be plugged into the 15-pin RPi connector as shown in the figure.

For the sake of simplicity, we will move ahead and discuss in detail the Infinite-ISP Titanium Ti180 Binary for IMX219 image sensor.

Infinite-ISP Binary file for Titanium Ti180 kit

Download the ‘Infinite-ISP_v1.5-IMX219.hex’ from the repo for flashing binaries on the FPGA kit (or check for latest release of binaries):Bringing Up the Titanium Development kit with Infinite-ISP Binaries:

Flashing the Infinite-ISP Binary file on Titanium Ti180 kit

You can skip to the next section if you are already familiar with the detailed process of flashing a binary file on the Titanium Ti180 kit using the Efinity IDE.

Now that you have selected the right binary file for the Titanium Ti180 kit, you need to properly flash the kit with the binary file and reset the Titanium Ti180 kit after flashing is complete to get the design running.

To flash the Titanium Ti180 kit with an Infinite-ISP binary, follow the aforementioned steps:1. Open Efinity IDE and then Open Efinity Programmer.

2. Change Programming Mode to ‘SPI Active using JTAG Bridge (New)’.

3. Update bitstream file path to program ‘Infinite-ISP_v1.5-IMX219.hex’ binaries and press start program button.

4. Reset the Titanium Ti180 board by turning it OFF and then ON and visualize the Infinite-ISP output on your monitor screen.

5. Connect the USB cable with the JTAG/USB port on Titanium Ti180 to read the messages over the serial interface.

Running Infinite-ISP on the Titanium Ti180 Kit

After flashing the Titanium Ti180 Kit with the relevant binary, attach the sensor to the Titanium Ti180 Kit. A power reset will load and run the Infinite-ISP binary on the Titanium Ti180 kit.

To access a user menu via serial interface, connect one end of the USB cable (USB type C) with the Titanium Ti180 Kit and the other end (USB A) to your PC.

Connect one end of the HDMI cable to the Titanium Ti180 Kit and the other end to a monitor/display. Insert the SD card into the SD card slot of the Titanium Ti180.

Plug in the power cable. The Infinite-ISP binary will start running on the Titanium Ti180 Kit. The details of the hardware connectivity are shown in the following figure.

The output of the Infinite-ISP will be displayed on the HDMI monitor.

To interact with the Infinite-ISP in real-time, a configuration menu over serial interface is also provided through which users can tune the Infinite-ISP, change the image sensor’s configuration such as exposure and gain settings in real-time and also capture RAW10 and processed frames in the SD card. This is demonstrated in the next section.

Live Demonstration

The effects of changing each of the configurations on ISP output can be explored and checked out in our demonstration of the latest Titanium FPGA ISP Binaries with IMX219 image sensor.

Conclusion

That wraps up our Getting Started guide for Infinite-ISP, the open-source image signal processor running on the Titanium Ti180 J484 Development Kit. Whether you're into camera systems or developing vision applications, you can hit the ground running with the pre-built FPGA binaries, quickly set up a camera system, tweak ISP and sensor parameters live to see their impact, and capture both RAW sensor data and processed output with ease.

In the next tutorial, we’ll show you how to build the Infinite-ISP FPGA project from source on the Titanium Ti180 J484 using the Efinity toolchain.

If you have any questions, suggestions, or feedback, please feel free to share them in the comments section below. We welcome your input and look forward to your insights.