Published January 19, 2022 © MIT

See around corners with a thermal camera!

With thermal imaging, glass acts as a mirror. Therefore, you can use picture frames and other glass objects to see reflections!

IntermediateFull instructions provided2 hours1,172

See around corners with a thermal camera!

Things used in this project

Hardware components

OpenMV Cam M7

FLIR Lepton FS

FLIR Lepton adaptor module

Software apps and online services

Edge Impulse Studio

OpenMV IDE

Story

I developed my first thermal camera app using a simple MLX90640 sensor that I got on sale and a Wio terminal that I already owned. After that, I was hooked. I loved the functionality but I wanted better resolution! I started looking into the FLIR Lepton modules, but they were pretty pricey ($250-300 USD). I found a solution in the Leptop FS, which cost $99 USD but has a little less functionality than the regular Leptons. I ended up also getting an Open MV H7 cam (the H7 Plus is out of stock until mid 2022) and an adaptor module. Still expensive, but you get 160x120 resolution as opposed to the 32x24 resolution of the MLX90640. And what a difference! The detail in the Lepton sensor is impressive. See the comparison photos below:

1 / 2 • MLX90640 sensor with 32x24 resolution

I was walking around my house with the Lepton and I noticed that I could see my reflection pretty clearly in windows and on glass picture frames and other glass objects. I did some googling and found out that infrared (IR) energy is reflected off of glass, much like visible light is reflected off of mirrors. A light bulb then went off. I wonder if I could set up the thermal camera to point towards a picture frame to covertly look around corners (or behind you). It turns out it works!

My setup. Notice the camera behind the monitor. This can see out the room and down the hall.

To add a little spin to it, I developed a person identification model using Edge Impulse to identify if a person is approaching. Very cool!

Success! This is the reflection off a picture frame. I'm down the hall outside the room.

There are plenty of Edge Impulse tutorials on here, so I won't go through the workflow of how I did it. It is essentially the same as other image classification projects, except I'm using thermal images instead of regular images. Unfortunately, because I have the regular H7 instead of the H7 Plus, I'm limited in the size of the model I can use. I'm hoping to try this out again with the H7 Plus with a larger model in the future.

This was the first time I used the EON Tuner in Edge Impulse, and I think that helped a lot in optimizing the model size and inference time.

Edge Impulse's EON Tuner found the optimized model for me!

With the OpenMV IDE and python, it's actually really easy to create a script that detects a person and then saves the image. See the code at the bottom of this project. You can then do whatever you want. If you have the Wifi shield, you could send an alert, or you could trigger an LED.

I hope you enjoyed this project. It was interesting to learn new things about how the non-visible light spectrum works! I look forward to continuing to explore this cool sensor!

Edge Impulse edited python script

# Edge Impulse - OpenMV Image Classification Example

import sensor, image, time, os, tf

sensor.reset()                         # Reset and initialize the sensor.
sensor.set_pixformat(sensor.GRAYSCALE)    # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA)      # Set frame size to QVGA (320x240)
sensor.set_windowing((240, 240))       # Set 240x240 window.
sensor.skip_frames(time=2000)          # Let the camera adjust.

net = "trained.tflite"
labels = [line.rstrip('\n') for line in open("labels.txt")]

clock = time.clock()
found = 0
if not "images" in os.listdir(): os.mkdir("images")
while(True):
    clock.tick()

    img = sensor.snapshot()

    # default settings just do one detection... change them to search the image...
    for obj in tf.classify(net, img, min_scale=1.0, scale_mul=0.8, x_overlap=0.5, y_overlap=0.5):
        print("**********\nPredictions at [x=%d,y=%d,w=%d,h=%d]" % obj.rect())
        img.draw_rectangle(obj.rect())
        # This combines the labels and confidence values into a list of tuples
        predictions_list = list(zip(labels, obj.output()))

        #person found!
        if predictions_list[1][1] > 0.75:
            img.draw_string(10,130,"Person found!")
            print("Person found!!!\n")
            file_name = "images/person_" + str(found) + ".jpeg"
            img = img.save(file_name)
            found = found+1
            print("Saved " + file_name)

    print(clock.fps(), "fps")
    time.sleep(1)