Fire Detection System
Jetson Board Software installation : Headless Device Mode
Machine Learning Algorithm
Putting all together
Payload Delivery System
Actioning the payload release system

Published March 5, 2020 © Apache-2.0

Drone Patrol

An autonomous drone equipped with an AI to detect fire on an early stage and deliver first-aid kits to victims.

IntermediateFull instructions providedOver 8 days5,557

Bonus Prizes

HoverGames Challenge 1: Fight Fire with Flyers

Things used in this project

Hardware components

NVIDIA Jetson Nano Developer Kit

Raspberry Pi Camera Module

SG90 Micro-servo motor

Wifi module

NXP RDDRONE-FMUK66

NXP KIT-HGDRONEK66

Software apps and online services

OpenCV

PX4 MAVLink

TensorFlow

Hand tools and fabrication machines

3D Printer (generic)

Story

There is no doubt that it is easier to extinguish a fire on an early stage before it was able to spread, therefore, the key to eliminating fires is a quick response of the emergency service. One of the most important improvements in fire safety is the development of methods for the early detection of fires and automatic monitoring of fire-hazardous areas.

Using NXP HoverGames drone development kit and QGroundControl software, I propose to build an autonomous drone capable of monitoring a defined area to detect fire beginning.

Drone patrols a given area autonomously around the clock according to a mission that has been pre-defined in QGroundControl. It utilizes an on-board camera and machine learning algorithms to detect fire hazards. When the drone detects a possible fire, the GPS coordinates are sent to the dispatch service desk.

The drone will have also a Payload flexibility : a 3D printed modular payload release system allows various devices to be attached to the drone. A search and rescue crew might attach a gas measurement device to monitor the air quality before entering in fire area or may release a map with instructions to assist trapped people in getting out of the fire zone or even delivering first-aid kits for victims.

Fire Detection System:

To be able to perform real time computer vision on the air, the flight controller unit can't provide enough computation power to execute machine learning algorithms. For this reason, we rely on a companion computer architecture.

Companion Computers can be used to interface and communicate with flight controller using the MAVLink protocol. By doing this, a companion computer gets all the MAVLink data produced by the autopilot (including GPS data) and can use it to make intelligent decisions during flight. There are two major parts to Companion Computers - hardware and software.

The Companion Computer hardware refers to the specific computer hardware being used. This is typically a small ARM-based Single Board Computer. For this project, I have used a Nvidia Jetson Nano Board.

The Companion Computer software refers to the programs and tools that run on the Companion Computer. They will take in MAVLink telemetry from the Flight Controller and can route and process the telemetry data. For this project, I have used DroneKit suite.

Hardware installation:

Nvidia Jetson Nano is a developer kit, which consists of a SoM (System on Module) and a reference carrier board. It is primarily targeted for creating embedded systems that need high processing power for machine learning, machine vision and video processing applications.

The Jetbot expansion board is a valuable add-on for Jetson nano Boards, It adopts 3x 18650 batteries with up to 12.6V voltage output. It provides protections for over-charge, over-discharge, over-current, and short circuit. An onboard APW7313 voltage regulator, provides stable 5V voltage for the Jetson Nano. A 0.91inch 128×32 pixels OLED, allows to display robot IP address, RAM, battery life, etc.

Jetbot expansion board

Jetson Nano Board attached to the drone and ready to fly

Raspberry Camera pointing Down

The Jetson Nano with Jetbot expansion board are atteched to the Drone Frame. The camera is pointing down.

Jetson Board Software installation : Headless Device Mode

For this project, we are running the Jetson Nano Developer Kit in a "headless" configuration. That means you do not hook up a monitor directly to the Jetson Nano Developer Kit. This method conserves memory resources on the Jetson Nano and has the added benefit of eliminating the requirement for extra hardware, i.e. a monitor, keyboard, and mouse.

In addition, we will further simplify the configuration by using "USB Device Mode". In this mode, Jetson Nano Developer Kit connects directly to a computer through an USB cable. This eliminates the need for a network connection on the Jetson Nano, as well as the need to determine the IP address on the network. It is always 192.168.55.1:8888 in this mode.

Download the NVIDIA DLI Jetson Nano SD Card Image
Download, install, and launch Etcher
Use Etcher to write the Jetson Nano Developer Kit SD Card Image to your microSD card.
Connect the Raspberry Pi v2 Camera to the MIPI CSI port
Reboot the Jetson Nano
Open the following link address : 192.168.55.1:8888The JupyterLab server running on the Jetson Nano will open up with a login prompt the first time.
Enter the password: dlinano

Launch a terminal session
Install required Python packages :

pip3 install tensorflow
pip3 install tflearn
pip3 install cv2

Machine Learning Algorithm:

For this project, I have used a Convolutional Neural Network (CNN). CNN is a type of neural network model which allows us to extract higher representations for the image content. Unlike the classical image recognition where you define the image features yourself, CNN takes the image’s raw pixel data, trains the model, then extracts the features automatically for better classification. The custom dataset used for training and evaluation can be found on Durham Collections (together with the trained network models). In addition, standard datasets such as furg-fire-dataset were also used for training and evaluation. You can download the python scirpts and the weights file needed to run the CNN with the following commands.

git clone https://github.com/amri1978/hovergames
cd hovergames/model
wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1KyIWAka3K31wE82R3jgIkKQE5FWWQhVw' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1KyIWAka3K31wE82R3jgIkKQE5FWWQhVw" -O 'fire.data-00000-of-00001' && rm -rf /tmp/cookies.txt

To test the fire detection CNN without all the stuff related to drone, you can use fire.py script (usage : python3 fire.py input_video_file -s). use the '-s' switch to save the output file. The output videos indicates when a fire is detected (red frame with FIRE word in the top left otherwise the frame is green and the word CLEAR is present).

Fire detection CNN architecture

If your are not familiar with CNN, stride and max pooling. You could read this excellent tutorial:

https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/

Putting all together:

To use fire detection CNN with HoverGames drone, connect Jetson Nano to FMU board with an USB cable.

Set a wifi USB module on the companion computer according to https://dev.px4.io/v1.9.0/en/qgc/video_streaming_wifi_broadcast.htmlIn QGroundControl, define a mission to scan the area to be monitored by droneDownload mission to the droneLaunch dronepatrolHeadless.py (you can add it to startup script to be launched when Jetson nano is powered on)The script is commented. The process is to :

Example of a mission in QgroundControl : the drone will follow waypoints

A fire alert message displayed in QGroundControl

There is another version of the script (dronepatrol.py) for desktop version of Jetson OS. I'have used this version to output demo videos below:

Long distance fire detection

Fire detection demo

Fire detection on fields

A funny example

Payload Delivery System :

The payload delivery system is based on a 3d print model and a small 9g servo (Hextronix HXT900 or equivalent).The 3d model consists of 4 parts:

- A bottom part to which the servo is attached- A piston or cylinder- A ring to attach the payload and blocked by the piston- A top part

The servo arm is attached to the piston, when the servo rotates counter-clockwise, it pulls the piston, making the ring free to drop and releasing the payload.

A 9g servo attached to the bottom part with the arm linked to piston (metal rigid wire) and the ring.

The payload release system fully mounted

A fire blanket attached to the payload release system

Actioning the payload release system:

The servo can be connected to auxiliary channel 5 or 6 and can be easily actioned using AUX Passthrough channels settings.

Schematics

Code

Drone Patrol Python code (Desktop version)

import cv2
import os
import sys
import math
import time

import tflearn
from tflearn.layers.core import *
from tflearn.layers.conv import *
from tflearn.layers.normalization import *
from tflearn.layers.estimator import regression

from dronekit import connect, VehicleMode, mavutil
from dronekit.mavlink import MAVConnection

def set_servo(servo_number, pwm_value):
    pwm_value_int = int(pwm_value)
    msg = vehicle.message_factory.command_long_encode(0,0,mavutil.mavlink.MAV_CMD_DO_SET_SERVO,0,servo_number, pwm_value_int,0,0,0,0,0)
    vehicle.send_mavlink(msg)

def my_new_fix_targets(message):
    pass

#return a CNN tensorflow model
def construct_fireCNN (x,y):

    network = tflearn.input_data(shape=[None, y, x, 3], dtype=tf.float32)

    network = conv_2d(network, 64, 5, strides=4, activation='relu')

    network = max_pool_2d(network, 3, strides=2)
    network = local_response_normalization(network)

    network = conv_2d(network, 128, 4, activation='relu')

    network = max_pool_2d(network, 3, strides=2)
    network = local_response_normalization(network)

    network = conv_2d(network, 256, 1, activation='relu')

    network = max_pool_2d(network, 3, strides=2)
    network = local_response_normalization(network)

    network = fully_connected(network, 4096, activation='tanh')
    network = fully_connected(network, 4096, activation='tanh')
    network = fully_connected(network, 2, activation='softmax')

    # constuct final model

    model = tflearn.DNN(network, checkpoint_path='fire',
                        max_checkpoints=1, tensorboard_verbose=2)

    return model



connection_string = "/dev/ttyUSB0" #connection to Jetson Nano USB port
baud_rate = 115200
print(">>>> Connecting with the UAV <<<")
vehicle = connect(connection_string,baud_rate, wait_ready=True)     #- wait_ready flag hold the program untill all the parameters are been read
udp_conn = MAVConnection('udpin:192.168.0.22:15667', source_system=1) #open an incoming udp connection on Jetson Nano
vehicle._handler.pipe(udp_conn)
udp_conn.master.mav.srcComponent = 1  # needed to make QGroundControl work!
udp_conn.start()
udp_conn.fix_targets = my_new_fix_targets







################################################################################

if __name__ == '__main__':

################################################################################
	

    while not vehicle.is_armable:
    	print("waiting to be armable")
    	time.sleep(1)

    print("Arming motors")
    vehicle.mode = VehicleMode("AUTO")
    vehicle.armed = True

    while not vehicle.armed: time.sleep(1)

    # construct and display model

    model = construct_fireCNN (224, 224)
    print("Constructed Fire CNN ...")

    model.load(os.path.join("model", "fire"),weights_only=True)
    print("Loaded CNN network weights ...")

################################################################################

    # network input sizes

    rows = 224
    cols = 224

    # display and loop settings

    windowName = "HoverGames Challenge 1: Fight Fire with Flyers with NXP";
    keepProcessing = True;

    # load video from default camera

    video = cv2.VideoCapture(0)
    print("Loaded video ...")

    # create window

    cv2.namedWindow(windowName, cv2.WINDOW_NORMAL);

    # get video properties

    width = int(video.get(cv2.CAP_PROP_FRAME_WIDTH));
    height = int(video.get(cv2.CAP_PROP_FRAME_HEIGHT))
    fps = video.get(cv2.CAP_PROP_FPS)
    frame_time = round(1000/fps);

    while (keepProcessing):

        # start a timer (to see how long processing and display takes)

        start_t = cv2.getTickCount();

        # get video frame from camera

        ret, frame = video.read()


        # re-size image to network input size and perform prediction

        small_frame = cv2.resize(frame, (rows, cols), cv2.INTER_AREA)
        output = model.predict([small_frame])

        # label image based on prediction

        if round(output[0][0]) == 1:
            cv2.rectangle(frame, (0,0), (width,height), (0,0,255), 50)
            cv2.putText(frame,'FIRE',(int(width/16),int(height/4)),
                cv2.FONT_HERSHEY_SIMPLEX, 2,(255,255,255),7,cv2.LINE_AA);
            msg = vehicle.message_factory.statustext_encode(1,b'Alert Fire at : ' + str( vehicle.location.global_frame).encode())
            udp_conn.master.mav.send(msg)
        else:
            cv2.rectangle(frame, (0,0), (width,height), (0,255,0), 50)
            cv2.putText(frame,'CLEAR',(int(width/16),int(height/4)),
                cv2.FONT_HERSHEY_SIMPLEX, 2,(255,255,255),7,cv2.LINE_AA);
            

        # stop the timer and convert to ms. (to see how long processing and display takes)

        stop_t = ((cv2.getTickCount() - start_t)/cv2.getTickFrequency()) * 1000;

        # image display and key handling

        cv2.imshow(windowName, frame);

        # wait fps time or less depending on processing time taken (e.g. 1000ms / 25 fps = 40 ms)

        key = cv2.waitKey(max(2, frame_time - int(math.ceil(stop_t)))) & 0xFF;
        if (key == ord('x')):
            keepProcessing = False;
        elif (key == ord('f')):
            cv2.setWindowProperty(windowName, cv2.WND_PROP_FULLSCREEN, cv2.WINDOW_FULLSCREEN);