Real-Time Video Stabilization Using Spresense

Intro

This project is based around the Spresense main board with the Spresense camera. Two video streams from the Spresense camera are shown on the TFT display. But one the stream is compensated for rotation/leaning of the Spresense board. The angle of rotation is calculated from the angle speed derived from a sensor add-on board containing a gyro which is mounted on top of the Spresense main board. A system like this could be used as a real-time stabilized camera.

System overview

There are four general parts in this project:

• Spresense main board
• Spresese camera
• MEMS Gyro/Accelerometer (Bosch BMI160 two board options listed in Hardware components section)
• Display

This application will set-up the camera to continuously fetch a video frame into a buffer. Very close in time to this event, the gyro data is read from the BMI160 sensor. The sensor will, among other data, provide the current angle speed in three axes. This data is provided with a precise timestamp. Using this timestamp together with the angle speed makes it is fairly easy to calculate the absolute angle. The Nuttx graphical system is used to define two rectangular windows on the display. One of the windows is filled with an unmodified (almost, it has been cropped and the pixel format has been changed) video stream from the camera buffer. The other window buffer is counter rotated using the absolute angle calculated from the gyro. The result of this is that two video streams are shown. In one window the video from the camera is displayed as the camera sees it. The other window shows a video that is always aligned, as if the camera is not rotating/rotated.

Hardware setup

Display

To diplay the data, a generic ILI9340 TFT display is used. The display is connected via SPI 5 on the main board. The main board GPIO voltage is only 1.8V which is sufficient for the ILI9340 display driver chip to recognize a logical 1 or 0.

Except for the regular SPI pins, SCK, MOSI and CS plus another two pins are used (RESET and DC). RESET line should speak for it self in regards of what the purpose of it is. DC is a "data or command" selection line. This line tells if the SPI transaction is a command to the TFT driver or if it is data. According to the ILI9340 specification the minimum Serial Clock cycle is 66ns which gives a maximum SPI clock of around 15MHz but in this example the SPI clock is driven to 45MHz with no apparent problems. However if it is driven faster than that, the output starts to look funny. The reason for this could be due to long wires between the Spresense board and the TFT display. Maybe the TFT driver chip could handle a higher clock speed.

Sensor

This project have been using a sensor add-on board for Spresense which have the Bosch BMI160 sensor mounted. This board might not yet be available in your region, but the same code have been tested using the BOOSTXL-SENSORS. If you test this with the BOOSTXL-SENSORS instead of the add-on board, the I2C slave address must be changed to 0x69 instead of 0x68. Details about this can be found in the BMI160 datasheet. The I2C slave address is defined in the file spresense/sdk/drivers/sensors/bmi160.c

$ sed -n 60p sdk/drivers/sensors/bmi160.c
#define BMI160_I2C_ADDR     0x69 /* If SDO pin is pulled to VDDIO, use 0x69 */

Changing the I2C slave address is not needed when using the add-on board. When using the BOOSTXL-SENSORS board, wires have to be used to connect the sensorboard to the Spresense main or extension board. Below is an image of the schematics when connecting the sensor board to the main board.

Software setup

This project have been developed using the NuttX based Spresense SDK which is more advanced than the Arduino IDE environment. The source files are located in the folder spresense/examples/bmi160_graphical in the build environment. The Spresense SDK is supported under Ubuntu on a PC.

Step by step instructions to build and flash the software:

$ git clone --recursive git@github.com:TE-KarlKomierowski/spresense.git
$ cd spresense
$ git checkout bmi_demo
$ cd sdk
$ tools/config.py -k release
$ make buildkernel
$ tools/config.py examples/bmi160_graphical
$ make
$ tools/flash.sh -c /dev/ttyUSB[x] -b 1152000 nuttx.spk #Change [x]

In the default configuration the TFT display is configured to be connected directly to the main board. This can be changed to connect to the extension board instead. This is done via menuconfig "make menuconfig" under Board Configuration->LCD SPI connection (Extension board: SPI4).

The serial output from the board will contain VT-100 terminal commands to make the data possible to read for a human. Minicom can be configured to support VT-100 terminal emulation. Start minicom and type Ctrl+z and then type T to enter the terminal settings. Pressing A will change Terminal Emulation. When you selected VT102 you should find the output stable, not scrolling and readable.

Settings

MEMS gyros tend to drift, defining the symbol USE_ACCEL_CALIB in bmi_graphical_main.c will also use the accelerometer (besides from the already used gyro) to reset the accumulated rotation variable once the rotation related to the g-vector can be calculated using the accelerometer. This code is not 100% perfect. It is mostly to show that it could be done.

Conclusion and potential future features

Gyros tend to drift. It is not trivial to find a way to compensate for this drift. While searching the internet I found numerous papers describing algorithms and methods on how to compensate for this. I actually didn't read any of them and in this project I use the accelerometer data to find the gravity vector and from that measure the angle of the board in three axes. This is of course not perfect but it can be a fun way to learn how it works playing around with various methods to get the output aligned as desired.

Potential future features/updates:

• Use the accelerometer to counter pan the image buffer to compensate for planar movements/shocks.
• Compensate for rotation around more than one axis.
• Use more than just one CPU core to handle the computation.
• Other suggestions?