Bilinear Image Scaling with FLIR Lepton & Minized

Introduction

Having recently (well about 5 months ago now) moved into a new office I wanted to create a little demo area to show case to potential clients. One of the things I have noticed over the years is that demos which have a visual element are always engaging.

Over several projects we have created image processing systems which operate in the visual wavelength on the Spartan Edge Accelerator and create a simple hand held remote thermal imager which uses a 32 pixel by 24 line sensor.

I wanted to created an application which uses a FLIR Lepton and along with a display connected to the MiniZed. This display would be a much higher resolution and enable me to demonstrate how we could scale up the image.

This project will therefore use the Avnet MiniZed connected to an Avnet 10 Inch touch display screen using the Pmod ports. I made this choice as the Spartan 7 edge demo uses a monitor, this means I can have both demonstrations going with only a single monitor. If you wanted to adapt the demo to work with a monitor the Pmod VGA connected to the Pmod ports.

Getting Started

The FLIR lepton is configured using an I2C interface, while the video is output over SPI using the video over SPI protocol. To keep within the export compliance rules the frame rate is 9 FPS and we need to synchronize to the video being output over the SPI interface.

We can capture this video using the SPI interface on the Zynq while also using the I2C to configure the device. To output the video we will be using a video processing pipeline in the programmable logic, this will do the following

• Read out the image from the PS DDR Memory
• Convert the image to a data stream
• Generate the video timing signals
• Convert the AXI Stream into parallel video timed in accordance with the timing signals
• Encode the data for transmission to the ten inch touch display.

To get started with this development we need to obtain the Avnet IP library from this library we need the Zed_Ali3_Controller.

Getting started with this we first need to create a new Vivado project which is targeting the MiniZed.

Once the project has been created the next step is to create a IP integrator block diagram and to add in the following IP

• Zynq processing system configured for the MiniZed
• AXI VDMA configured for Read operation
• Video Timing controller configured for generation only
• AXIS Subset controller to convert the VDMA stream from 32 to 24 bits
• AXIS Stream to Video out IP block
• AXI Quad SPI Interface
• Zed Ali3 Controller

The clocks for the system are as follows

• AXI Stream Clock - 200 MHz
• Pixel Clock - 71.1 MHz
• AXI Clock - 125 MHz

The complete block diagram looks as below

Block Configurations are

VDMA

VTC

AXI Subset

AXI Stream to Video Out

AXI Quad SPI

ALI3 Controller

We can use IP integrators connection automation wizard to implement the necessary processor reset blocks.

The constraints for the project are

set_property PACKAGE_PIN M10 [get_ports {ss_o   }];  # "M10.ARDUINO_IO10"
set_property PACKAGE_PIN R11 [get_ports {io1_i}];  # "R11.ARDUINO_IO12"
set_property PACKAGE_PIN P11 [get_ports {sck_o   }];  # "P11.ARDUINO_IO13"

set_property IOSTANDARD LVCMOS33   [get_ports {ss_o   }];  # "M10.ARDUINO_IO10"
set_property IOSTANDARD LVCMOS33   [get_ports {io1_i}];  # "R11.ARDUINO_IO12"
set_property IOSTANDARD LVCMOS33   [get_ports {sck_o   }];  # "P11.ARDUINO_IO13"

set_property PACKAGE_PIN G15 [get_ports {IIC_0_scl_io}];  # "G15.I2C_SCL"
set_property PACKAGE_PIN F15 [get_ports {IIC_0_sda_io}];  # "F15.I2C_SDA"

set_property IOSTANDARD LVCMOS33  [get_ports {IIC_0_scl_io}];  # "G15.I2C_SCL"
set_property IOSTANDARD LVCMOS33  [get_ports {IIC_0_sda_io}];  # "F15.I2C_SDA"

set_property PACKAGE_PIN P14 [get_ports {ALI3_ali_data_n[0]}];  # "M15.PMOD1_D0_N"
set_property PACKAGE_PIN P13 [get_ports {ALI3_ali_data_p[0]}];  # "L15.PMOD1_D0_P"
set_property PACKAGE_PIN N12 [get_ports {ALI3_ali_data_n[1]}];  # "M14.PMOD1_D1_N"
set_property PACKAGE_PIN N11 [get_ports {ALI3_ali_data_p[1]}];  # "L14.PMOD1_D1_P"
set_property PACKAGE_PIN R15 [get_ports {ALI3_ali_data_n[2]}];  # "L13.PMOD1_D2_N"
set_property PACKAGE_PIN P15 [get_ports {ALI3_ali_data_p[2]}];  # "K13.PMOD1_D2_P"
set_property PACKAGE_PIN R13 [get_ports {ALI3_ali_data_n[3]}];  # "N14.PMOD1_D3_N"
set_property PACKAGE_PIN R12 [get_ports {ALI3_ali_data_p[3]}];  # "N13.PMOD1_D3_P"


set_property PACKAGE_PIN M15 [get_ports {ALI3_ali_clk_n     }];  # "M15.PMOD1_D0_N"
set_property PACKAGE_PIN L15 [get_ports {ALI3_ali_clk_p     }];  # "L15.PMOD1_D0_P"
set_property PACKAGE_PIN K13 [get_ports {ALI3_ali_rst_n     }];  # "K13.PMOD1_D2_P"

set_property IOSTANDARD TMDS_33   [get_ports {ALI3_ali_data_n[0]}];  # "M15.PMOD1_D0_N"
set_property IOSTANDARD TMDS_33   [get_ports {ALI3_ali_data_p[0]}];  # "L15.PMOD1_D0_P"
set_property IOSTANDARD TMDS_33   [get_ports {ALI3_ali_data_n[1]}];  # "M14.PMOD1_D1_N"
set_property IOSTANDARD TMDS_33   [get_ports {ALI3_ali_data_p[1]}];  # "L14.PMOD1_D1_P"
set_property IOSTANDARD TMDS_33   [get_ports {ALI3_ali_data_n[2]}];  # "L13.PMOD1_D2_N"
set_property IOSTANDARD TMDS_33   [get_ports {ALI3_ali_data_p[2]}];  # "K13.PMOD1_D2_P"
set_property IOSTANDARD TMDS_33   [get_ports {ALI3_ali_data_n[3]}];  # "N14.PMOD1_D3_N"
set_property IOSTANDARD TMDS_33   [get_ports {ALI3_ali_data_p[3]}];  # "N13.PMOD1_D3_P"
set_property IOSTANDARD TMDS_33  [get_ports {ALI3_ali_clk_n}];  # "P14.PMOD2_D0_N"
set_property IOSTANDARD TMDS_33  [get_ports {ALI3_ali_clk_p}];  # "P13.PMOD2_D0_P"
set_property IOSTANDARD LVCMOS33 [get_ports {ALI3_ali_rst_n}];  # "P15.PMOD2_D2_P"

With the application created in Vivado the next step is to implement the software design.

The next step is to generate a software application using Vitis.

Vitis Software Development

With the hardware available the next step is to import the XSA into the Vitis and create the platform. We can then create an application to run on the platform, the first thing we need to do if we wish to use the UART is to set the BSP to use UART 1 for STD IN and OUT.

Once that is created we need to do the following in out application

• Initialize the SPI / I2C / VTC / VDMA Components
• Configure the I2C interface to configure the FLIR Lepton
• Configure the VDMA to output a frame of 1280 by 800 to fit the 10 inch touch display
• Configure the video timing controller to generate timing for the 1280 by 800 video output
• Read out frames from the Lepton using SPI synchronized to the frame rate

The output from the VoSPI is a 80 pixel by 60 line image the raw grey scale output which is 14 bits across two bytes.

As we are displaying the video on a 24 bit RGB interface, we need to provide 8 bits of per colour channel. As such each 14 bit word is truncated to 8 bits to be able to fit in the pixels. To generate a greyscale image we will set each color channel to the same pixel value. Of course this does reduce the dynamic range of the pixel.

Once the image has been captured in an array we need to scale up that image for the output display.

The first method I tried and had used previously with the FLIR was to just output each pixel in the same a number of times depending on a simple scale factor.

This simple approach would ensure the synchronization could be achieved for the 9 frames per second the imager outputs.

The output for this is

frameBuf[iPixelAddr] = (u32) (((Image[(y/scaley)][(x/scalex)+4])<<16)|										((Image[(y/scaley)][(x/scalex)+4])<<8)|										((Image[(y/scaley)][(x/scalex)+4])));

This does scale up the image and work however, the output looks blocky as can be seen in the images below

A better approach would be to interpolate between the know pixel values, to scale the image. This however requires more computational power, we can use the video multi scaler in the programmable logic however, for this project I want to demonstrate the principals of up scaling an image and not just include a IP block where the implementation is hidden.

Linear interpolation is pretty straight forward, between two points we can interpolate a value of any point between the two points as shown below with the equation

x = A (B - x)/(B-A) + B(x-A)/(B-A)

Of course when we want to scale an image up we have not only 2 points we want to scale up but a two dimensional array.

This is more complicated as we need to consider four elements not two

We can determine the value of Z using the equation

z = A(1−(x-x1/x2-x1))(1−(y-y2/y2-y1))+B(x-x1/x2-x1)(1−yy)+C(1−(x-x1/x2-x1))(y-y2/y2-y1)+D(x-x1/x2-x1)(y-y2/y2-y1)

Implementing this in the software

x_l = floor(scalex *x);
y_l = floor(scaley *y);
x_h = ceil(scalex *x);
y_h = ceil(scaley *y);

x_weight = (scalex * x) - x_l;
y_weight = (scaley * y) - y_l;

a = Image[y_l][x_l];
b = Image[y_l][x_h];
c = Image[y_h][x_l];
d = Image[y_h][x_h];

pixel = a * (1- x_weight) * ( 1 - y_weight) +
	b * x_weight * (1 - y_weight) +
	c * y_weight * (1 - x_weight) +
	d * x_weight * y_weight;

The value of this pixel can then be written to the output buffer. Of course this time we are using several complex math instructions for each pixel in the frame. We need to be able to do these fast in the application.

As such we need to make sure the code can run as fast as possible on the processor. For this reason we need to se the code optimization to O3 and remove any debugging elements.

Doing this results enables synchronization to be maintained and image to be output correctly with a much better resizing.

Wrap Up

The MiniZed demonstrates it capabilities well as it shows the ability to maintain sync with the FLIR Lepton and output the images scaled up very nicely using a bilinear interpolation for the pixels.

The code is of course on my github