Gesture Controlled Drone: Part 2

In the last part in this series for controlling a drone using gestures, I showed you how to use the the gesture sensor onboard the Arduino Nano 33 BLE Sense to get your drone to take off, land and move left or right.

The problem with using just the gesture sensor is that the sensor is able to read only 4 gestures, so apart from taking off and landing, we could only map two other gestures (which I mapped to moving right or left). We could use a combination of the gesture sensor and the IMU to control our drone better.

In this project, I will add support for direction recognition using the IMU to the previous project to increase the range of motion of our gesture control for the drone.

Software

Just like before, we need to do two steps to finish this project: Recognize gestures and IMU movements, and then send those details to the drone so that it can move. The first step is to add IMU to our previous gesture recognition script:

Gesture Recognition and IMU

The LSM9DS1 is the sensor that you can use to take inertial measurements (IMU). In addition to that, it also has a 3D accelerometer, gyroscope and magnetometer, so you can use it for a lot of applications. Here is the code to do that:

#include <Arduino_APDS9960.h>
#include <Arduino_LSM9DS1.h>

float x, y, z;

int plusThreshold = 90, minusThreshold = -90;

void setup() {
  Serial.begin(9600);
  pinMode(LED_BUILTIN, OUTPUT);
  pinMode(LEDR, OUTPUT);
  pinMode(LEDG, OUTPUT);
  pinMode(LEDB, OUTPUT);

  while (!Serial);
  if (!APDS.begin()) {
    Serial.println("Error Initialising sensor");
  }

  Serial.println("Detecting Gestures");
  digitalWrite(LEDR, HIGH);
  digitalWrite(LEDG, HIGH);
  digitalWrite(LEDB, HIGH);

  if (!IMU.begin()) {
    Serial.println("Failed to initialize IMU!");
  }

  Serial.print("Gyroscope sample rate = ");
  Serial.print(IMU.gyroscopeSampleRate());
  Serial.println(" Hz");
  Serial.println();
  Serial.println("Gyroscope in degrees/second");

}

void loop() {
  if (IMU.gyroscopeAvailable()) {
    IMU.readGyroscope(x, y, z);
  }

  if(y > plusThreshold)
  {
    Serial.println("Moving back");
    delay(2000);
  }
  if(y < minusThreshold)
  {
    Serial.println("Moving front");
    delay(2000);
  }
  if(x < minusThreshold)
  {
    Serial.println("Moving left");
    delay(2000);
  }
    if(x > plusThreshold)
  {
    Serial.println("Moving right");
    delay(2000);
  }
    if (APDS.gestureAvailable()) {
    // a gesture was detected, read and print to serial monitor
    int gesture = APDS.readGesture();

    switch (gesture) {
      case GESTURE_UP:
        Serial.println("Detected UP gesture");
        digitalWrite(LEDR, LOW);
        delay(1000);
        digitalWrite(LEDR, HIGH);
        break;

      case GESTURE_DOWN:
        Serial.println("Detected DOWN gesture");
        digitalWrite(LEDG, LOW);
        delay(1000);
        digitalWrite(LEDG, HIGH);
        break;

      default:
        // ignore
        break;
    }
  }
}

In the code, I use only two gestures: Up and Down for take-off and landing. I use the gyroscope to measure 4 directions: front, back, right and left. I hold the board in my left hand with the micro-usb connection towards the left of my fist. You could hold the board in a different orientation and might have to change the direction. You can do that either in the Arduino sketch above or in the python script below.

Sending Commands to the Drone

Now that we can measure the gestures and gyroscope direction, we need to send those commands to the drone. We do that using a Python script that reads the serial output from the drone and uses the DJITelloPy package to send those commands to the drone.

Before that, you will have to install python packages to control the drone and read from the serial output:

pip install djitellopy pyserial

Before running the code (below), you need to connect your laptop/pi to the drone's wifi which will look like TELLO-XXXXXX.

from djitellopy import Tello
import time
import serial

ser = serial.Serial('/dev/cu.usbmodem141401')
ser_bytes = ser.readline()
ser.flushInput()

tello=Tello()

tello.connect()
print(tello.get_current_state())
print('Starting')

flying=False

start=time.time()
while True:
    try:
        ser_bytes = ser.readline()
        decoded_bytes = str(ser_bytes[0:len(ser_bytes)-2].decode("utf-8"))
        print(decoded_bytes)
        if 'UP' in decoded_bytes and not flying:
            tello.takeoff()
            print("Takeoff Done")
            flying=True
        elif 'DOWN' in decoded_bytes and flying:
            tello.land()
            flying=False
            print("Drone Landed")
        elif 'left' in decoded_bytes and flying:
            tello.move_down(40)
            print("Moved Left")
        elif 'right' in decoded_bytes and flying:
            tello.move_up(40)
            print("Moved Right")
        elif 'front' in decoded_bytes and flying:
            tello.move_right(50)
            print("Moved Front")
        elif 'back' in decoded_bytes and flying:
            tello.move_left(50)
            print("Moved Back")
        if time.time()-start>60:
            tello.land()
            flying=False
            print("Landing Drone")
    except:
        print("Keyboard Interrupt")
        tello.land()
        break

The drone will move 50 cm when moving left and right and 40cm when going up or down. You can customize these values to suit your needs.

For safety reasons (and also because I tested this indoors in my home), I have included some safety features that you can toggle:

• To prevent any random UP gestures from causing the drone to take-off, the drone will only take-off on the second gesture (which has to be UP. The first gesture can be anything). You can remove this by removing these lines in the beginning: ser_bytes = ser.readline();ser.flushInput()
• The drone will automatically land after 60 seconds of receiving the first gestures. You can remove this or increase the flight time by removing the second if statement in the while loop
• The drone will land if you shut down the python script. THIS IS IMPORTANT and you should NOT REMOVE this since there is no way to control a flying drone if you exit out of the program.

Hardware

The hardware connections you need to make to get this to work is simple: Connect the Arduino to your laptop or Raspberry Pi which you need to connect to the drone via WIFI. Below is a schematic:

Flight Demonstration

Future Work

This is not a complete solution and more features can be added to it. Here are some ideas I have for future versions:

• Using the gyroscope means that you can only control the direction and not the speed or acceleration of the drone. I want to use the onboard Accelerometer to have even more fine-grained control of the drone.
• Since the Nano does not have WIFI onboard we currently need a wifi capable device in addition to it. I want to remove that dependency by using a Nano RP2040 instead that has wifi onboard.

I am currently working on these and you can check out the code or contribute in github here.

Full Code and Questions

You can find the full code here under blogs/part_2. If you have any questions, leave a comment here, or create an issue on github!