Published © MIT

Autonomous AI Assessment & Communications Platform

AI enhanced drone assisting firefighters on-the-ground & remote command and control centers via IR & HD streams and gas sensor data

AdvancedFull instructions providedOver 2 days2,267

Things used in this project

Hardware components

Coral Dev Board
Google Coral Dev Board
×1
Coral Environmental Sensor Board
Google Coral Environmental Sensor Board
×1
Google Coral Dev Board Camera
×1
FLIR Miniature IR USB Camera w/Lepton 3.0 module
×1
Seeed Studio Grove O2 Sensor
×1
Seeed Studio Grove Gas Sensor (MQ9)
For Carbon Monoxide (CO) detection
×1
Seeed Studio Grove CO2 Sensor
×1
PixyCam2
For automatic landing system
×1
Seeed Studio Grove TF Mini Lidar
For automatic landing system
×1
Pan-Tilt HAT
Pimoroni Pan-Tilt HAT
For high-def and IR camera orientation
×1
RDDRONE-FMUK66
NXP RDDRONE-FMUK66
×1
KIT-HGDRONEK66
NXP KIT-HGDRONEK66
×1

Software apps and online services

Robot Operating System
ROS Robot Operating System
PX4
PX4
QGroundControl
PX4 QGroundControl

Hand tools and fabrication machines

3D Printer (generic)
3D Printer (generic)

Story

Read more

Custom parts and enclosures

FLIR IR Camera Mount

Custom mount for our FLIR IR camera for the Pimoroni PanTiltHat

Pimoroni PanTiltHat mount

Mount for the Pimoroni PanTilthat to fit onto the drone dual rails

Coral Dev Board with Sensor Board mount

3D printed case for the Coral Dev board with the Sensor board attached. This is the top part of the case and is a modification of the Coral Dev board case by doctorjay found on Thingiverse at https://www.thingiverse.com/thing:3561461.

We used the bottom part of the above 3D model with our customized top section.

Code

drone python web server

Python
This is the python code for the drone's web page that streams live video & sensor data and allows remote control of the pan/tilt for the cameras. We created a systemd service file and ran this directly from it.
#!/usr/bin/env python3

# Copyright 2020 James Ewing jim@droidifi.com
#
#Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
#
#1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
#
#2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
#
#3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
#
#THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# code developed for the Hovergames Fight Fire with Flyers contest at http://www.hackster.io
#
import threading
import argparse
import itertools
import os
import json

import numpy as np
from imutils.video import VideoStream
import cv2
from streamer import Streamer

import seeed_sgp30
from grove.i2c import Bus

import pantilthat
from sys import exit

from coral.enviro.board import EnviroBoard
from coral.cloudiot.core import CloudIot
from luma.core.render import canvas
from PIL import ImageDraw
from time import sleep

from flask import Flask, render_template, Response, request, copy_current_request_context, send_from_directory
from flask_socketio import SocketIO, emit, disconnect

try:
    sgp30 = seeed_sgp30.grove_sgp30(Bus())
except:
    sgp30 = None
    
try:
    enviro = EnviroBoard()
except:
    enviro = None
    
sensors = {}
threads = {}
delay = 3000
display_duration = 5
sensor_data_thread = None
sensor_thread_lock = threading.Lock()
sensor_data_lock = threading.Lock()
async_mode = None
namespace = '/sensor_data'

class IRVideoCamera(object):
    def __init__(self):
        self.image_res = (160, 120)
        self.video_src = 1
        self.video = cv2.VideoCapture(self.video_src)
    
    def __del__(self):
        self.video.release()
    
    def get_frame(self):
        success, image = self.video.read()
        ret, jpeg = cv2.imencode('.jpg', cv2.resize(image, self.image_res),
            params=(cv2.IMWRITE_JPEG_QUALITY, 70))
#        ret, jpeg = cv2.imencode('.jpg', image)
        return jpeg.tobytes()
    
class HDVideoCamera(object):
    def __init__(self):
        self.image_res = (160, 120)
        self.video_src = 0
        self.video = cv2.VideoCapture(self.video_src)
    
    def __del__(self):
        self.video.release()
    
    def get_frame(self):
        success, image = self.video.read()
        image = ~image  # invert image
        ret, jpeg = cv2.imencode('.jpg', cv2.resize(image, self.image_res),
            params=(cv2.IMWRITE_JPEG_QUALITY, 70))
#        ret, jpeg = cv2.imencode('.jpg', image)
        return jpeg.tobytes()
    
class people_mapper:
    def __init__(self, time_spotted, lat, lng):
        self.time_spotted = time_spotted
        self.lat = lat
        self.lng = lng
        
    people = ()
    people_by_time = {people.time: person for person in people}
    

def update_display(display, msg):
    with canvas(display) as draw:
        draw.text((0, 0), msg, fill='white')

app = Flask(__name__)
socketio = SocketIO(app, async_mode=async_mode)

@app.route('/', defaults={'path': ''})
@app.route('/<path:path>')
def catch_all(path):
    return render_template('index.html', async_mode=socketio.async_mode)

@app.route('/js/<path:path>')
def send_js(path):
    return send_from_directory('js', path)

def web_server():
#    host = os.environ.get('SERVER_HOST', 'localhost')
#    try:
#        port = int(os.environ.get('SERVER_PORT' '80'))
#    except:
#        port = 80
    app.run(host="0.0.0.0", port=80, threaded=True)

@app.route('/api/<direction>/<int:angle>')
def api(direction, angle):
    if angle < 0 or angle > 180:
        return "{'error':'out of range'}"

    angle -= 90

    if direction == 'pan':
        pantilthat.pan(angle)
        return "{{'pan':{}}}".format(angle)

    elif direction == 'tilt':
        pantilthat.tilt(angle)
        return "{{'tilt':{}}}".format(angle)

    return "{'error':'invalid direction'}"
        
def gen(camera):
    while True:
        frame = camera.get_frame()            
        yield (b'--frame\r\n'
               b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n\r\n')

@app.route("/ir_camera")        
def ir_camera():
    return Response(gen(IRVideoCamera()),
                    mimetype='multipart/x-mixed-replace; boundary=frame')        

def gen2(camera):
    while True:
        frame = camera.get_frame()            
        yield (b'--frame\r\n'
               b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n\r\n')
        
@app.route("/hd_camera")        
def hd_camera():
    return Response(gen2(HDVideoCamera()),
                    mimetype='multipart/x-mixed-replace; boundary=frame')   

def update_sensors_thread():
    global sensors
    count = 0
    while True:
        print('sending sensor data')
        socketio.sleep(30)
        update_sensor_data()
        print('sending: ' + json.dumps(sensors));
        socketio.emit('update', sensors, namespace=namespace)
        
@socketio.on('connect', namespace=namespace)
def sensor_data_connect():
    global sensor_data_thread, sensor_thread_lock
    print('client connected')
    with sensor_thread_lock:
        if sensor_data_thread is None:
            sensor_data_thread = socketio.start_background_task(update_sensors_thread)
#    print('send')
    get_sensor_data()

@socketio.on('get_data', namespace=namespace)
def get_sensor_data():
    global sensors
    print('received get_sensor_data')
    update_sensor_data()
    print('sending: ' + json.dumps(sensors));
    emit('update', sensors , namespace=namespace)
    
socketio.on('disconnect_request', namespace=namespace)
def disconnect_request():
    @copy_current_request_context
    def can_disconnect():
        disconnect()
        
@socketio.on('disconnect', namespace=namespace)
def client_disconnect():
    print('client disconnected', request.sid);
    
def _none_to_zero(val):
    return float(0.0) if val is None else round(val,2)

def update_sensor_data():
    global sensors
    co2_eq_ppm = 0.0 
    tvoc_ppb = 0.0
    o2_pct = 0.0
    with sensor_data_lock:
        try:
            co2_data = sgp30.read_measurements()
            co2_eq_ppm, tvoc_ppb = co2_data.data
        except:
            print('could not update CO2 & VoL values')
        try:
            o2_pct = enviro.grove_analog
            o2_pct = (o2_pct * (5.0 / 1023.0) * 0.21 / 2.0) * 100.0
        except:
            print('could not update O2 value')
        
        sensors['temperature'] = _none_to_zero(enviro.temperature)
        sensors['humidity'] = _none_to_zero(enviro.humidity)
        sensors['ambient_light'] = _none_to_zero(enviro.ambient_light)
        sensors['pressure'] = _none_to_zero(enviro.pressure)
        sensors['co2'] = _none_to_zero(co2_eq_ppm)
        sensors['vol'] =_none_to_zero(tvoc_ppb)
        sensors['o2'] = _none_to_zero(o2_pct)
        sensors['co'] = _none_to_zero(0.0)
    
def read_sensors():
    global sensors
    read_period = int(delay / (2 * display_duration))
    update_sensor_data()
    for read_count in itertools.count():
        msg = 'Temp: %.2f C\n' % sensors['temperature']
        msg += 'RH: %.2f %%' % sensors['humidity']
        update_display(enviro.display, msg)
        sleep(display_duration)
        msg = 'VoC: %.2f ppm\n' % sensors['vol']
        msg += 'CO2: %.2f ppm' % sensors['co2']
        update_display(enviro.display, msg)
        sleep(display_duration)
        msg = 'O2: %.2f %%\n' % sensors['o2']
        msg += 'Pressure: %.2f kPa' % sensors['pressure']
        update_display(enviro.display, msg)
        sleep(display_duration)
    
if __name__ == '__main__':
    threads["sensors"] = threading.Thread(target=read_sensors)
    threads["web_server"] = threading.Thread(target=web_server)

    threads["sensors"].start()
    threads["web_server"].start()
    
    threads["sensors"].join()
    threads["web_server"].join()
    
    socketio.run(app)

drone python flask template

HTML
This is the html template find templates/index.html
<!doctype html>
<html lang="en">
    <head>
        <meta name="charset" value="utf-8">
        <title>FFWF Camera Pan Tilt</title>
        <style type="text/css">
            .main {
                width:160px;
                height:160px;
                background:#F0F0F0;
                position:relative;
            }
            .main ul, .main li {
                list-style:none;
                margin:0;
                padding:0;
            }
            .main li {
                width:35px;
                height:35px;
                border:5px outset #DDDDDD;
                background:#CCCCCC;
                position:absolute;
                cursor:pointer;
                line-height:35px;
                text-align:center;
            }
            .main .up {left:50%;margin-left:-25px;}
            .main .down {left:50%;bottom:0;margin-left:-25px;}
            .main .left {top:50%;margin-top:-25px;}
            .main .right {top:50%;right:0;margin-top:-25px;}
        </style>
	<script src="/js/jquery-3.4.1.slim.min.js" integrity="sha256-pasqAKBDmFT4eHoN2ndd6lN370kFiGUFyTiUHWhU7k8=" crossorigin="anonymous"></script>
        <script type="text/javascript" src="/js/socket.io.slim.min.js"></script>
        <script type="text/javascript" charset="utf-8">            
        $(document).ready(function() {
            var namespace = '/sensor_data';
            var socket = io(namespace); //io.connect('http://' + document.domain + ':' + location.port + namespace);
            
           socket.on('connect', function(){
                console.log('connected');
                socket.emit('get_data');
           });
            
           socket.on('update', function(msg){
                console.log('receiving sensor data');
                console.log(msg);
                
                var sensor_list = document.getElementById('sensor_list');
                while(sensor_list.firstChild) sensor_list.removeChild(sensor_list.firstChild);                

                var entry;
                entry = document.createElement('li');
                entry.appendChild(document.createTextNode('Temperature: ' + msg.temperature + ' C'));
                sensor_list.appendChild(entry);
                
                entry = document.createElement('li');
                entry.appendChild(document.createTextNode('Light: ' + msg.ambient_light + ' lux'));
                sensor_list.appendChild(entry);
                
                entry = document.createElement('li');
                entry.appendChild(document.createTextNode('Pressure: ' + msg.pressure + ' kPa'));
                sensor_list.appendChild(entry);
                
                entry = document.createElement('li');
                entry.appendChild(document.createTextNode('Humidity: ' + msg.humidity + ' %'));
                sensor_list.appendChild(entry);
                
                entry = document.createElement('li');
                entry.appendChild(document.createTextNode('Carbon Dioxide: 400 ppm'));
                sensor_list.appendChild(entry);
                
                entry = document.createElement('li');
                entry.appendChild(document.createTextNode('Oxygen: ' + msg.o2 + ' %'));
                sensor_list.appendChild(entry);
                
                entry = document.createElement('li');
                entry.appendChild(document.createTextNode('Volatile Organics: ' + msg.vol + ' ppm'));
                sensor_list.appendChild(entry);
                
//                 if(cb)
//                     cb();
            });
        });
        </script>
    </head>
    <body>
        <h1>Fight Fire With Flyers</h1>
    
        <img src="{{ url_for('ir_camera') }}"/><img src="{{ url_for('hd_camera') }}"/>
        <ul id="sensor_list">      
        </ul>       
        <div class="main">
            <ul>
                <li class="up" data-action="pan:+1">Up</li>
                <li class="down" data-action="pan:-1">Down</li>
                <li class="left" data-action="tilt:-1">Left</li>
                <li class="right" data-action="tilt:+1">Right</li>
            </ul>
        </div>        
        <script
            src="/js/jquery-3.1.1.min.js"
            crossorigin="anonymous"></script>
        <script type="text/javascript">
        
            var current_pan = 90;
            var current_tilt = 90;
            var pantilt_speed = 60; // Delay between increments in ms

            var interval = null;
            var current_direction = null;
            var current_angle = null;

            $(function(){
                $(window).on('keydown',function(e){
                    clearInterval(interval);

                    switch(e.keyCode){
                        case 38: // Arrow Up
                            current_direction = 'tilt';
                            current_angle = 1;
                            interval = setInterval(pantilt,pantilt_speed);
                            break;
                        case 40: // Arrow Down
                            current_direction = 'tilt';
                            current_angle = -1;
                            interval = setInterval(pantilt,pantilt_speed);
                            break;
                        case 37: // Arrow Left
                            current_direction = 'pan';
                            current_angle = 1;
                            interval = setInterval(pantilt,pantilt_speed);
                            break;
                        case 39: // Arrow Right
                            current_direction = 'pan';
                            current_angle = -1;
                            interval = setInterval(pantilt,pantilt_speed);
                            break;
                    }
                });
                $(window).on('keyup',function(e){clearInterval(interval)});
                $('.main').on('mousedown','li',function(e){
                   e.preventDefault();
                   clearInterval(interval);

                   var obj = $(this);
                   var action = obj.data('action');
                   current_direction = action.split(':')[0];
                   current_angle = parseInt(action.split(':')[1]);

                   interval = setInterval(pantilt,pantilt_speed);
               })
               .on('mouseup','li',function(e){
                   clearInterval(interval);
               });

               function pantilt(){
                   var angle = 0;

                   if(current_direction == 'pan'){
                       current_pan += current_angle;
                       if(current_pan < 0) current_pan = 0;
                       if(current_pan > 180) current_pan = 180;
                       angle = current_pan;
                   }

                   if(current_direction == 'tilt'){
                       current_tilt += current_angle;
                       if(current_tilt < 0) current_tilt = 0;
                       if(current_tilt > 180) current_tilt = 180;
                       angle = current_tilt;
                   }

                   $.get('/api/' + current_direction + '/' + angle); 
                }
            });
        </script>
    </body>
</html>

Hovergames PX4 repository

Link to our PX4 repository with code for the Pixelcam2 automatic landing system

Credits

Comments

Related channels and tags