SmarTC - Smart Thermal Camera

Overview

It was May 2017 and I bumped into an internet page about a challenge organized by Flir and BeagleBone.org Foundation. I know Flir cameras very well and it was clear in my head how they can be useful for robotic applications. The joint of Flir and BeagleBone Blue SBC was the better starting point to put into practice what was in my mind and the project “SmarTC – Smart Thermal Camera” has born.

"As a Computer Engineer specialized in Robotics, Computer Vision and Artificial Intelligence I would like to use the FLIR Lepton 3 LWIR camera to create an intelligent sensor, capable of better understanding the environment where a robot is operating.Using a LWIR sensor the robot could better distinguish the obstacles in the environment and detect “animated” (hot) and “inanimate” (cold) objects, reacting to their movements, according to their classification. The sensor could also be used to detect humans and animals making the robots safest.Furthermore the LWIR sensor makes the robot capable to operate safely also in lack of light, during night hours or during in an electrical fault.The combination of the FLIR Lepton 3 LWIR sensor and the computational power of the BeagleBone Blue board can so result in an “smart thermal camera” useful for autonomous domestic robots or autonomous flying drones."

I submitted my idea for the Challenge and I crossed my fingers.

The time passed and it was the 1st August when I received an email by the Judging Panel of the Challenge telling me that I was one of the five finalists of the contest. In a few days I received a Flir Lepton3 thermal camera and a BeagleBone Blue SBC and I started to work on the project.I must say that it was not really easy to start to handle the Lepton3 and I faced a lot of troubles to write a stable C/C++ code to acquire thermal images with the BeagleBone Blue, but with the help of previous works made on the former Lepton sensors and a lot of patience of Jason Kridner (the father of the Beagle SBC family) I finally reached my goal: I had a thermal video stream in my hand.

Flir Lepton 3 interface is made of two buses: SPI tocontinuously transmit the thermal frames, I2C to receive control command and to send theinformation about the status of the sensor. Everything is well described in two documents: “FLIR LEPTON® 3 Long Wave Infrared (LWIR) Datasheet” and “FLIR LEPTON® Software IDD“. Understanding the contents of both documents in detail is necessary to deal with Lepton3… I understood this statement after 10 days of panic.

The SmarTC project won the challenge: https://lepton.flir.com/news-and-updates/lepton-3-beaglebone-blue-contest-winner/

Demo Video

The steps of the work that I made can be summed up with the following YouTube videos:

And the final video, with the presentation of the basis of my algorithm to be applied to Robotics is this: