Published October 31, 2016 © Apache-2.0

HandTalk A Smart Handglove Interpreter

This project works as an Interpreter, converts hand gestures into text or voice or in commands.

IntermediateShowcase (no instructions)2 days16,549

Things used in this project

Hardware components

Arduino Nano R3

STMicroelectronics Flex Sensor

ProtoCentral Electronics ProtoCentral Pulse Oximeter & Heart Rate Sensor based on MAX30100

GSM Module SIM 900A

RF Module

For Wireless Operation

Adafruit Standard LCD - 16x2 White on Blue

Buzzer

BC 547

LED (generic)

Software apps and online services

Arduino IDE

Story

Introduction

The HANDTALK glove needs to be worn on the hand by the disabled and depending on the variation of the movements, the device will convert it intelligently into beep (can be voice too) and in a text. This message will be out from the beeper and LCD display. The HANDTALK glove senses the movements through the flex sensors which detect the different patterns of motion. The device can sense carefully each resistance and each movement by the hand. Currently the device can convert only few, but depending on the success of this device few more additional features maybe added later onto this expressive system.

Technology has always been of great help to the disabled and given them a helping hand to allow them to live a normal and healthy life like others.

Embedded systems are designed to do some specific task, rather than be a general purpose computer for multiple tasks. “Embedded” reflects the fact that they are an integral part of the system. Some also have real time performance constraints that must be met, for reason such as safety and usability; others may have low or no performance requirements, allowing the system hardware to be simplified to reduce costs. An embedded system is not always a separate block- very often it is physically built-in to the device it is controlling.

Today, embedded systems are found in cell phones, digital cameras, camcorders, portable video games, calculators, and personal digital assistants, microwave ovens, answering machines, home security systems, washing machines, lighting systems, fax machines, copiers, printers etc.

An embedded system is a system that has software embedded into hardware, which makes a system dedicated for an application (s) or specific part of an application or product or part of a larger system. It processes a fixed set of pre-programmed instructions to control electro-mechanical equipment which may be part of an even larger system.

Working:

We are working on this project namely Hand talk: Interpreter for the Differently Abled that translates hand gestures into a particular message. Flex Sensors fixed with the glove pick up the gesture made by the individual and then with the help of Arduino, that analog input is converted into digital for various gestures. For every particular gesture there is specific digital output and that digital output for specific gesture is reserved for specific message. So, when the individuals do that particular gesture, the predefined message for that gesture is displayed on the LCD along with a beep (sound) and that specific message is also transmitted in the form of text or call with the help of GSM module to the various registered numbers.

For instance, if the patient perform a gesture of fist (most often we people form the fist in case of fear, emergency), there would be a message displayed on the LCD: “CRITICAL SITUATION” or “EMERGENCY” along with a beep and that particular message would also be transmitted in the form of text or call to the Doctor’s phone or the Relative’s one.

Not only can this but the particular needy person also use this glove to control the lights, fan, AC or any other devices, yes! We are talking about home automation by gestures i.e. controlling of devices smartly by various gestures.

The main advantage with this design was its simplicity and the cheap components and this project can help to improve greatly the communication barrier between disabled people and the regular world.

CIRCUIT OPERATION

This Project has two parts :

1) Transmitter Section (Hand Glove)

2) Receiver Section (Display Module)

Transmitter Section :

This section is the main part of this project. It consists hand glove (fitted with Flex sensors and Heartbeat module) ,GSM module ,RF Transmitter and Arduino Uno as a Controller. All the gestures made by the hand glove (with Flex sensor ) is converted into different messages and commands for the operation of different devices .Example , if i want to start a fan then a command is send to the receiver end by the microcontroller with the help of RF module and the fan is easily controlled by the corresponding gestures. Now in case, i need water (or wheelchair or anything else) , all I need to do is to make a corresponding gesture accordingly and that gesture would be converted into text message and sent to the other person through the GSM module . Now using the Android app we can convert the text messages into voice also.

The Heartbeat module is used for the Heart rate monitoring of the patient .If there is any problem in heart rate then a message is also sent to the doctor or family members also.

In this project currently we are using seven gestures due to the problem of the accuracy but we can add about 100 plus gestures. Some examples of gestures are in shown in images.

Receiver Section:

This section is only for the display the message or the operation of different devices through the gesture. RF Receiver receives the command and this command is catch by the Arduino and Arduino controls the actions of this sections. In this part we are using fewer devices but you can also interface more devices. In this section LED bulb, motor and beeper is used for showing different action by the different gestures.

LCD (16x2) is used for the displaying the messages or requirement of the patients at different places .We can also place this section in the doctors chambers also.

Code

Arduino Code

//Code is written by Sachin Verma,Student of Dronacharya College of enginnering,Gr. Noida,India).Contact vsachin094@gmail.com.....Kindly don't copy this code //

#include <LiquidCrystal.h>
LiquidCrystal lcd(12,11,10,9,8,7);
int flexPin1 = A0;
int flexPin2 = A1;
int flexPin3 = A2;
int flexPin4 = A3;
int value1; 
int value2;
int value3;
int value4;
int timesTosend=1;
int count=0;
void setup()
{ 
   lcd.begin(16,2);
    lcd.setCursor(0,0);
    lcd.print("Welcome");
  Serial.begin(9600);      
  Serial.println("AT+CMGF=1"); //set GSM to text mode
  pinMode(6, OUTPUT);
  digitalWrite(6, LOW); // buzzer
}

void loop(){
  
  value1 = analogRead(flexPin1);        
  value2 = analogRead(flexPin2);  
  value3 = analogRead(flexPin3);  
  value4 = analogRead(flexPin4);  
 Serial.print(value1);
  
  value1 = map(value1, 700, 900, 0, 255);
  value2 = map(value2, 700, 900, 0, 255);
  value3 = map(value3, 700, 900, 0, 255);
  value4 = map(value4, 700, 900, 0, 255);
  
  if(value1 >80 && value2 <70 && value3 <70 && value4 <70)
  {
    Serial.print("Food");
    lcd.clear();
    lcd.print("Food");
    digitalWrite(6, HIGH);
    delay(500);
    digitalWrite(6, LOW);
          while(count<timesTosend){
          delay(1500);
          Serial.print("AT+CMGS=\"");
          Serial.print("90690461xx");
          Serial.println("\"");
          while (Serial.read()!='>');
          {
            Serial.print("Food Requirement by patient");  //SMS body
            delay(500);
            Serial.write(0x1A);  // sends ctrl+z end of message
            Serial.write(0x0D);  // Carriage Return in Hex
            Serial.write(0x0A);  // Line feed in Hex
            //The 0D0A pair of characters is the signal for the end of a line and beginning of another.
            delay(5000);
          }
          count++;
          }  }
  if(value1 >80 && value2 >80 && value3 <70 && value4 <70)
  {
     lcd.clear();
    lcd.print("Medicine");
    Serial.print("Medicine");
    digitalWrite(6, HIGH);
    delay(500);
    digitalWrite(6, LOW);
          while(count<timesTosend)
{
          delay(1500);
          Serial.print("AT+CMGS=\"");
          Serial.print("80105595xx");
          Serial.println("\"");
          while (Serial.read()!='>');
          {
            Serial.print("Medicine Requirement by patient");  //SMS body
            delay(500);
            Serial.write(0x1A);  // sends ctrl+z end of message
            Serial.write(0x0D);  // Carriage Return in Hex
            Serial.write(0x0A);  // Line feed in Hex

            //The 0D0A pair of characters is the signal for the end of a line and beginning of another.
            delay(5000);
          }
          count++;
          }  
  }
  if(value1 >80 && value2 >80 && value3 >80 && value4 <70)
  {
     lcd.clear();
    lcd.print("Wheelchair");
    Serial.print("Wheelchair");
    digitalWrite(6, HIGH);
    delay(500);
    digitalWrite(6, LOW);
          while(count<timesTosend){
          delay(1500);
          Serial.print("AT+CMGS=\"");
          Serial.print("90690461xx");
          Serial.println("\"");
          while (Serial.read()!='>');
          {
            Serial.print("Wheelchair Requirement by patient");  //SMS body
            delay(500);
            Serial.write(0x1A);  // sends ctrl+z end of message
            Serial.write(0x0D);  // Carriage Return in Hex
            Serial.write(0x0A);  // Line feed in Hex
            //The 0D0A pair of characters is the signal for the end of a line and beginning of another.
            delay(5000);
          }
          count++;
          }
  }
  if(value1 >80 && value2 >80 && value3 >80 && value4 >80)
  {
     lcd.clear();
    lcd.print("Normal Condition");
    Serial.print("Nothing Condition");
  }  
  if(value1 <70 && value2 <70 && value3 <70 && value4 <70)
  {
     lcd.clear();
    lcd.print("Critical");
    Serial.print("Critical");      
    digitalWrite(6, HIGH);
    delay(500);
    digitalWrite(6, LOW);
     Serial.print("ATD + +9190690461xx;");
          delay(100);
          Serial.println();
          delay(30000);            // wait for 30 seconds...
          Serial.println("ATH");   // hang up
          while(count<timesTosend){
          delay(1500);
          Serial.print("AT+CMGS=\"");
          Serial.print("80105595xx");
          Serial.println("\"");
          while (Serial.read()!='>');
          {
            Serial.print("Emergency Call");  //SMS body
            delay(500);
            Serial.write(0x1A);  // sends ctrl+z end of message
            Serial.write(0x0D);  // Carriage Return in Hex
            Serial.write(0x0A);  // Line feed in Hex
            //The 0D0A pair of characters is the signal for the end of a line and beginning of another.
            delay(5000);
          }
          count++;
          }    
  }
  if(value1 <70 && value2 >80 && value3 >80 && value4 >80)
  {
     lcd.clear();
    lcd.print("Water");
    Serial.print("Water");
    
    digitalWrite(6, HIGH);
    delay(500);
    digitalWrite(6, LOW);
          while(count<timesTosend){
          delay(1500);
          Serial.print("AT+CMGS=\"");
          Serial.print("90690461xx");
          Serial.println("\"");
          while (Serial.read()!='>');
          {
            Serial.print("Water Requirement by patient");  //SMS body
            delay(500);
            Serial.write(0x1A);  // sends ctrl+z end of message
            Serial.write(0x0D);  // Carriage Return in Hex
            Serial.write(0x0A);  // Line feed in Hex
            //The 0D0A pair of characters is the signal for the end of a line and beginning of another.
            delay(5000);
          
          }}