Step One: Planning and Setting the Objectives
Step Two: Use Cases and Design
Step Three: Overview of Process
Step Four: Implementation
Step Five: Calculations Involved and their Implementation
Step Six: Testing
Future Insight

Published September 6, 2022

Smart Gesture and Voice Control for Display Devices IoT

Take the freedom of operating with display devices into your hands and voice commands; eliminating the hassle of finding "the remote"!

IntermediateProtip1,463

Smart Gesture and Voice Control for Display Devices IoT

Things used in this project

Hardware components

Espressif ESP8266 ESP-12E

IR receiver (generic)

Bipolar - RF Transistor, NPN

Ultrasonic Sensor - HC-SR04 (Generic)

LED (generic)

ams OSRAM Osram SFH 4545 IR LEDs

Software apps and online services

IFTTT SmartThings service

Grandeur

Story

Is it not aggravating for you to look for your device’s remote control, when your favourite show is on air? Is it delightful for you to interrupt or delay your favourite video game and spend a significant amount of time trying to find the controls on your device’s remote? Would it not be captivating for you to control your LED at home or any electronic display device at workplace on the command of your hand gestures and voice control? We know right! That is something we have all wanted to happen at some point in our lives. We bring to you the freedom to control your display devices by mere voice commands and even simpler, hand gestures.

Step One: Planning and Setting the Objectives

The first step in making any project valuable is to plan the project ahead and set clear and precise objectives, which saves time and proves productive. In this regard, the path was set using a conceptual diagram. A conceptual diagram serves like a mind-map and eases to convey the objectives to others. Following is the conceptual diagram:

Concept Diagram-Made with Wondershare EdrawMax

Step Two: Use Cases and Design

After identifying the objectives, next step is to formulate and test the Use Cases. Furthermore, the design of the project is to be finalized. Following are one of the few Use Cases:

Innovative Approach: An innovative approach is to be aimed at to revolutionize the concept of control for display devices.
Resource Management: The resources are used efficiently so that the project can be replicated at the ease of availability and affordable cost, while making the best out of used resources.
Data Management: The data is managed and transferred securely, to ensure reliability.
Process Optimization and Efficiency: The data is gathered efficiently by controlling access to the physical range of the detecting device to make better use of physical space.

"Design is thinking made visual." -Saul Bass

The design of the device is simple yet efficient enough to allow continuous operation and make the experience worthwhile for user.

Step Three: Overview of Process

The process of making the device to control the display devices is simple yet technical. The process of making the project includes four main aspects:

1. Choosing and Connecting the hardware

2. Programming the hardware

3. Integration of hardware with Online Services and Hand Tools

4. Approach towards making

Choosing and Connecting the hardware: The choice of right hardware is the first step towards making any great project. The hardware should be user-friendly and efficient. The list of hardware is mentioned in the section above. However, IR Receiver was initially used for analysis purposes. It is considered optional.

The hardware is connected as follows:

Connected Chosen Hardware

Programming the hardware: The hardware is programmed using ArduinoIDE, which is an open-source, cross-platform and easy-to-use platform to program the board or microcontroller.

Integration of hardware with Online Services: The Online Services such as Google Home and Google Assistant are used to implement the voice control feature. Furthermore, IFTTT(If This Then That) is implemented to connect the devices and services in order to achieve automation. The platform allows a trigger to set a specific data value. Moreover, Grandeur provides effective interaction with the data.

Approach towards making: The approach varies for gesture and voice control, except for the common use of ESP8266 and software. The gesture control uses Ultrasonic sensor and IR Transmitter LED, whereas voice control uses a series of operations, which are explained in detail in coming steps.

Step Four: Implementation

The implementation is based on the four main aspects as discussed in previous step. The roadmap of implementation is visualized as follows:

Workflow Diagram-Made with Wondershare EdrawMax

Voice Control: The voice control makes use of the following:

Google Assistant
Google Home
IFTTT
Grandeur
ESP8266
IR Transmitter LED

The voice command being received by Google Assistant uses Google Home; which is connected using the same account with IFTTT, which in turn sends the data to Grandeur. Grandeur interacts with the data and through its services of interaction with data, provides guaranteed delivery of data to receivers; which in this case is ESP8266, which in turn transmits data to IR Transmitter LED. The software code is uploaded to the ESP8266 microcontroller. The IR Transmitter LED communicates with the display device as commanded.

Following is the tutorial to set up and integrate IFTTT and Grandeur to manipulate the commands from Google Assistant.

1 / 12 • Step1: Create the Applet

Gesture Control: The gesture control is made possible by the following:

Ultrasonic Sensor
IR Transmitter LED
ESP8266

The motion is detected by the ultrasonic sensors, which measure the distance or displacement using ultrasonic waves by calculating the time difference between emission and reception, with significant efficiency.

Initially, the object; which in this case is hand, is detected. After 10 milliseconds, the position of the hand movement is calculated. If the movement is upwards, the velocity is positive; otherwise negative. The limit of 0.001 meter per second is set for the changing of channel and changing the levels of volume. One ultrasonic sensor is used for changing the channel, while the other is used for volume control. Then, the data is passed to the IR Transmitter LED to execute the command, via ESP8266.

For changing the channel, if the hand is moved up, the device changes to previous channel and if the hand is moved downward, the the device changes to next channel. Similarly, analogous to this, the volume increases on upward hand movement and otherwise for downward hand movement.

Step Five: Calculations Involved and their Implementation

The calculations are only involved for the gesture control in which displacement and velocity are measured to detect the gesture direction, to distinguish among the commands.

Source: learntocalculate.com

VerticalVelocity: The vertical velocity is calculated separately in a unique function.

double verticalVelocity(int sensorNo)
{
  currentDist[sensorNo] = verticalDistance(sensorNo);
  currentTime[sensorNo] = millis();

  if(currentDist[sensorNo] > 3 && currentDist[sensorNo] < 50 && currentDist[sensorNo] != previousDist[sensorNo])
  {
    interval[sensorNo] = previousTime[sensorNo] - currentTime[sensorNo];
    if(interval[sensorNo] < 1000)
    {
      velocity[sensorNo] = ( double(previousDist[sensorNo] - currentDist[sensorNo]) / interval[sensorNo] ) / 10;
      previousDist[sensorNo] = currentDist[sensorNo];
      previousTime[sensorNo] = currentTime[sensorNo];
      return velocity[sensorNo];
    }
    else
    {
      previousDist[sensorNo] = currentDist[sensorNo];
      previousTime[sensorNo] = currentTime[sensorNo];
      return 0;
    }
  }
  return 0;
}

HorizontalVelocity: Similarly, the horizontal velocity is also handled as follows:

double horizontalVelocity()
{
  int distA = distanceSensorA.measureDistanceCm();
  int distB = distanceSensorB.measureDistanceCm();

  if(timeNA == false && timeNB == false)
  {
    if(distA < 50 && distA > 0)
    {
      timeA = millis();
      timeNA = true;
    }
    else if (distB < 50 && distB > 0)
    {
      timeB = millis();
      timeNB = true;
    }
  }
  else if(timeNA == true && timeNB == false)
  {
    if (distB < 50 && distB > 0)
    {
      timeB = millis();
      timeNB = true;
    }
  }
  
  else if(timeNA == false && timeNB == true)
  {
    if (distA < 50 && distA > 0)
    {
      timeA = millis();
      timeNA = true;
    }
  }
  else
  {
    timeNA = false;
    timeNB = false;
    double inter = double(timeB - timeA) / 1000;
    if(abs(inter) > 0.075 && abs(inter) <= 0.450)
      return (0.107 / inter);
  }
  return 0;
}

Step Six: Testing

The most important step before deployment of any project is to test it on various conditions and record the observations. The Smart Control for Display Devices was also tested rigorously.

Voice Control Testing

Channel Control

Volume Control

Gesture Control Testing

Gesture Control

Grandeur Canvas

Successfully Tested Voice Commands

Future Insight

In a nutshell, the article provides a deep understanding about the Smart Control for Display Devices. Moreover, it also delineates the complete procedure of how you can make it, even with a little previous knowledge about electronics.

The general opinion about technology is that it has made life easier and human relationships difficult. However, we believe that technology, if used smartly; can restore the essence in human relationships. The project can be used on various platforms ranging from educational to entertainment purposes. Furthermore, the future for this device seems exquisite, if used on a large scale for the greater good

Code

Code

/* Including the SDK and WiFi library */
#include <Grandeur.h>
#include <ESP8266WiFi.h>
#include <IRremoteESP8266.h>
#include <IRrecv.h>
#include <IRutils.h>
#include <IRsend.h>
#include <HCSR04.h>

/* Configurations */
String deviceID = "devicel7uetwtp00190kgiegd3f0oc";
String apiKey = "grandeurl7ueq9b600140kgi8dwv165j";
String token = "320d185132d2c303371a1ff6aa42fdee8b4006bd9f3bdaca2f3efcdbdfc93b25";

/* WiFi credentials */
String ssid = "YOUR WIFI SSID";
String password = "PASSWORD";

/*Variables*/
int LED1 = 0;
int LED2 = 0;

const int trigPin[2] = {D6, D8};  //D4
const int echoPin[2] = {D5, D7};  //D3

UltraSonicDistanceSensor distanceSensorA(trigPin[0], echoPin[0]);  // Initialize sensor that uses digital pins 13 and 12.
UltraSonicDistanceSensor distanceSensorB(trigPin[1], echoPin[1]);

int previousDist[2] = {20, 20};
int previousTime[2] = {millis(), millis()};

int currentDist[2];
int currentTime[2];

int interval[2];
double velocity[2];

int timeA = millis();
int timeB = millis();

bool timeNA = false;
bool timeNB = false;

double statVeloVA = 0;
double statVeloVB = 0;
double statVeloH = 0;

// An IR detector/demodulator is connected to D2 (on a NodeMCU board).

const uint16_t kIrLed = D2;  // ESP GPIO pin to use. Recommended: 4 (D2)
IRsend irsend(kIrLed);  // Set the GPIO to be used to sending the message.
decode_results results;


/* Create variable to hold project and device */
Grandeur::Project project;
Grandeur::Project::Device device;

/* Function prototypes */
void onConnection(bool status);
void nextUpdateHandler(const char* path, const char* state);
void prevUpdateHandler(const char* path, const char* state);
void upUpdateHandler(const char* path, const char* state);
void downUpdateHandler(const char* path, const char* state);
void connectWiFi();
int verticalDistance(int sensorNo);
double verticalVelocity(int sensorNo);
double horizontalVelocity();

void setup() {
    /* Begin the serial */
    Serial.begin(9600);

    /* Connect to WiFi */
    connectWiFi();

    /* Initializes the global object "grandeur" with your configurations. */
    project = grandeur.init(apiKey, token);

    /* Get reference to device */
    device = project.device(deviceID);

    /* Sets connection state update handler */
    project.onConnection(onConnection);

    /* Add event handler on state variable */
    device.data().on("next", nextUpdateHandler);
    device.data().on("prev", prevUpdateHandler);
    device.data().on("up", upUpdateHandler);
    device.data().on("down", downUpdateHandler);

    /* Set mode of LED to output */
    pinMode(LED_BUILTIN, OUTPUT);
    pinMode(16, OUTPUT);

    /* Turn the LED off by default */
    digitalWrite(LED_BUILTIN, 1);
    digitalWrite(16, 1);
  
    pinMode(2, OUTPUT);
    digitalWrite(2, 1);
    pinMode(16, OUTPUT);
    digitalWrite(16, 1);
    
    pinMode(trigPin[0], OUTPUT); // Sets the trigPin as an Output
    pinMode(echoPin[0], INPUT); // Sets the echoPin as an Input
    
    pinMode(trigPin[1], OUTPUT); // Sets the trigPin as an Output
    pinMode(echoPin[1], INPUT); // Sets the echoPin as an Input
    
    irsend.begin();

}

void loop() {
    /*  Runs the SDK.
        Connects to Grandeur only after the device is connected to WiFi.
    */
    if(WiFi.status() == WL_CONNECTED) project.loop();

    double veloH = 0, veloVA = 0, veloVB = 0;
    
    if(millis() % 45 == 0)
      veloVB = verticalVelocity(1);
    else if(millis() % 47 == 0)
      veloVA = verticalVelocity(0);

    int distA = distanceSensorA.measureDistanceCm();
    int distB = distanceSensorB.measureDistanceCm();
    
    if(statVeloVA != veloVA && veloVA != 0)
    {
      Serial.printf("Distance A = %d cms\n", distA);
      Serial.printf("Distance B = %d cms\n", distB);
      Serial.printf("Vertical Velocity A = %f m/s\n", veloVA);
      Serial.println();
      if(veloVA >= 0.001)
      {
        Serial.print("Activated Next Channel: ");
        Serial.println(1);
        Serial.println();
        
        irsend.sendNEC(0xFF52AD);
      }
      else if (veloVA <= -0.001)
      {
        Serial.print("Activated Previous Channel: ");
        Serial.println(1);
        Serial.println();
        
        irsend.sendNEC(0xFF12ED);
      }
      statVeloVA = veloVA;
    }

    if(statVeloVB != veloVB && veloVB != 0)
    {
      Serial.printf("Distance A = %d cms\n", distA);
      Serial.printf("Distance B = %d cms\n", distB);
      Serial.printf("Vertical Velocity B = %f m/s\n", veloVB);
      Serial.println();
      statVeloVB = veloVB;
      if(veloVB >= 0.001)
      {
        Serial.print("Activated Volume Up: ");
        Serial.println(1);
        Serial.println();
        
        irsend.sendNEC(0xFF30CF);
      }
      else if (veloVB <= -0.001)
      {
        Serial.print("Activated Volume Down: ");
        Serial.println(1);
        Serial.println();
        
        irsend.sendNEC(0xFF18E7);
      }
    }
}

/* Function to check device's connection status */
void onConnection(bool status) {
    switch(status) {
        case CONNECTED:
            /* Device connected to the cloud */
            
            Serial.println("Device is connected to the Grandeur.");
            
            return;

        case DISCONNECTED:
            /* Device disconnected from cloud */

            Serial.println("Device is disconnected from the Grandeur.");
            return;
    }
}

/* Function to handle state update event */
void nextUpdateHandler(const char* path, const char* state) {

    /* Print state */
    Serial.printf("\nLast Voice Command Received on %s\n", state);
    Serial.printf("Command was change to Next channel! \n");
    LED1 = !LED1;
    /* Update pin level */
    digitalWrite(LED_BUILTIN, !LED1);
    irsend.sendNEC(0xFF52AD);
}

/* Function to handle state update event */
void prevUpdateHandler(const char* path, const char* state) {

    /* Print state */
    Serial.printf("\nLast Voice Command Received on %s\n", state);
    Serial.printf("Command was change to Previous channel! \n");
    LED2 = !LED2;
    /* Update pin level */
    digitalWrite(16, !LED2);
    irsend.sendNEC(0xFF12ED);
}

/* Function to handle state update event */
void upUpdateHandler(const char* path, const char* state) {

    /* Print state */
    Serial.printf("\nLast Voice Command Received on %s\n", state);
    Serial.printf("Command was TV Volume Up! \n");
    LED1 = !LED1;
    /* Update pin level */
    digitalWrite(LED_BUILTIN, !LED1);
    irsend.sendNEC(0xFF30CF);
}

/* Function to handle state update event */
void downUpdateHandler(const char* path, const char* state) {

    /* Print state */
    Serial.printf("\nLast Voice Command Received on %s\n", state);
    Serial.printf("Command was TV Volume Down! \n");
    LED2 = !LED2;
    /* Update pin level */
    digitalWrite(16, !LED2);
    irsend.sendNEC(0xFF18E7);
}

/* Function to connect to WiFi */
void connectWiFi() {
    /* Set mode to station */
    WiFi.mode(WIFI_STA);

    /* Connect using the ssid and password */
    WiFi.begin(ssid, password);

    /* Block till the WiFi is connected */
    while (WiFi.status() != WL_CONNECTED) {
            delay(500);
            Serial.print(".");
    }

    /* Print message */
    Serial.println("");
    Serial.println("WiFi connected");

    /* And IP address */
    Serial.println(WiFi.localIP());
}


int verticalDistance(int sensorNo)
{ 
  
  long duration;
  int dis;
  // Clears the trigPin
  digitalWrite(trigPin[sensorNo], LOW);
  delayMicroseconds(2);
  
  // Sets the trigPin on HIGH state for 10 micro seconds
  digitalWrite(trigPin[sensorNo], HIGH);
  delayMicroseconds(10);
  digitalWrite(trigPin[sensorNo], LOW);
  
  // Reads the echoPin, returns the sound wave travel time in microseconds
  duration = pulseIn(echoPin[sensorNo], HIGH);
  
  // Calculating the distance
  dis = duration / 29 / 2;

  return dis;
}

double verticalVelocity(int sensorNo)
{
  currentDist[sensorNo] = verticalDistance(sensorNo);
  currentTime[sensorNo] = millis();

  if(currentDist[sensorNo] > 3 && currentDist[sensorNo] < 50 && currentDist[sensorNo] != previousDist[sensorNo])
  {
    interval[sensorNo] = previousTime[sensorNo] - currentTime[sensorNo];
    if(interval[sensorNo] < 1000)
    {
      velocity[sensorNo] = ( double(previousDist[sensorNo] - currentDist[sensorNo]) / interval[sensorNo] ) / 10;
      previousDist[sensorNo] = currentDist[sensorNo];
      previousTime[sensorNo] = currentTime[sensorNo];
      return velocity[sensorNo];
    }
    else
    {
      previousDist[sensorNo] = currentDist[sensorNo];
      previousTime[sensorNo] = currentTime[sensorNo];
      return 0;
    }
  }
  return 0;
}

double horizontalVelocity()
{
  int distA = distanceSensorA.measureDistanceCm();
  int distB = distanceSensorB.measureDistanceCm();

  if(timeNA == false && timeNB == false)
  {
    if(distA < 50 && distA > 0)
    {
      timeA = millis();
      timeNA = true;
    }
    else if (distB < 50 && distB > 0)
    {
      timeB = millis();
      timeNB = true;
    }
  }
  else if(timeNA == true && timeNB == false)
  {
    if (distB < 50 && distB > 0)
    {
      timeB = millis();
      timeNB = true;
    }
  }
  
  else if(timeNA == false && timeNB == true)
  {
    if (distA < 50 && distA > 0)
    {
      timeA = millis();
      timeNA = true;
    }
  }
  else
  {
    timeNA = false;
    timeNB = false;
    double inter = double(timeB - timeA) / 1000;
    if(abs(inter) > 0.075 && abs(inter) <= 0.450)
      return (0.107 / inter);
  }
  return 0;
}