Why tackle the Smart Home problem
What’s new
Why did we decide to make it
Working
Circuit Diagram
Hardware and Equipment used:
Implementation:
Videos
Useful links:

Team Blitzkrieg: Mantej Gill, Divyansh, RITESH MUKHOPADHYAY

Published March 27, 2017 © Apache-2.0

J.A.R.V.I.S. : A Virtual Home Assistant

J.A.R.V.I.S. is a personal home automation assistant for controlling electrical home appliances integrated with an augmented reality app.

AdvancedFull instructions provided20 hours116,522

Things used in this project

Hardware components

Arduino 101

Arduino 4 Relays Shield

Arduino GSM shield V2

Software apps and online services

Unity

Android Studio

Vuforia

Adobe Illustrator

Arduino IDE

Hand tools and fabrication machines

Soldering iron (generic)

Hot glue gun (generic)

Story

Why tackle the Smart Home problem?

Nowadays, people have smartphones with them all the time. So it makes sense to use these to control home appliances. Presented here is a home automation system using a simple Android app which you can use to control electrical appliances with voice commands. Using the app you can control all the appliances like TV, fans, light etc. It’s like a Siri which can be controlled using voice (command given by user). You can give the command to switch on or off the devices (like light, fan etc) and also manipulate them like fan speed, light intensity. Commands are sent via a Bluetooth module to Arduino. So there is no need for you to get up to switch on or switch off the device while watching a movie or doing some important work.

What’s new?

We are also using an AR-augmented reality app (self made) to control the appliances and manipulate them. If we want to switch on the electrical appliance we just need to focus the camera and it will start it and using a virtual button we can switch it off .

If we want to know the electricity unit consumed by our appliances along with the cost of units we just need to send a text message to the device (Arduino) through our phone. The Arduino would be connected to the network through a GSM module which would reply with another text message containing the electricity unit and the total cost consumed by the appliances. This will help the user to get update on the electricity bill and they can decide whether to use the appliances on a regular routine or reduce it usage to save electricity.

Why did we decide to make it?

Working

The Digital Energy Meter

In the energy meter, a LDR (Light Dependent Resistor) is used to take the blink speed according to the load. The blink per second value is fed into the Arduino for further calculation units and the value of energy consumed. These are sent to the user via the GSM module.The user can then comprehend which appliance is consuming more power and take action accordingly.

1 / 2

The AI Virtual Assistant

The AI Virtual Assistant works using an android app which is installed on the user’s android phone. The user activates the agent using the wake up command, ‘Listen up Jarvis’.

The android app is connected to the Arduino 101 via Bluetooth which is inturn connected to the relay and hence the electrical appliances.

The flow of signals is as follows:

1. Voice Input to Android App

2. Processing on Android App

3. Sending encrypted data to Arduino via Bluetooth.

4. Processing on Arduino 101

5. Digital signals to relay

Pins connections:

Pin 8 -- > PulseIn [Digital Input]

Pin 6 -- > IN1 -- > Fan Output

Pin 4 -- > IN2 -- > Light Output

Pin 5 -- > IN3 -- > Coffee Maker

1 / 5

1 / 2

Connections of Energy meter:

Connect the meter to the supply.

Cover the energy meter with a black tape, preventing the LDR inside the energy meter from outside light.

Connect the neutral wire directly to the loads and the phase wire to the relay.

Take the wires from the LDR outside the meter to the counter circuit.

Now take the output from the counter circuit to the assigned pin 8 pulsepin as in the code.

Connect the TX, RX pin of the Arduino 101 to RX, TX pin of the GSM module.

Now power up the boards after burning the code.

Circuit Diagram

Circuit Diagram Of Energy meter and AI Virtual Assistant

Hardware and Equipment used:

Inside Energy meter: LDR Light Dependent Resistor

GSM module

Counter circuit: potentiometer, IC LM358N

Arduino 101

BlueTooth-HC05 module

4 channel 12V 10A relay control board module with optocoupler for PIC AVR ARM

Programming Platforms

· For programming the Arduino 101, we have used the default Arduino IDE.

· We have developed our Android app using Android Studio (JAVA and XML).

· Database connectivity is done using Google's Firebase.

· For augmented reality we have used Unity along with the Vuforia SDK.

Implementation:

All the components are giving readings to the Arduino, where we are converting these readings into character code in order to send the bytes efficiently and quickly to the computer. Inside a never ending loop, every reading corresponds to a character.

A code should run on the computer in order to receive the data that is being sent by arduino via serial ports.

For developing the android app we are using android studio and to convert voice to text we are using the standard google api.

For speech recognition we are using acoustic-phonetic speech parameters for speaker-independent speech recognition in the android app.

We have the AI (artificial intelligence) virtual assistant which uses deep learning techniques like CNN (Convolutional Neural Network) using python to implement the natural language processing.

For the augmented reality app we are using Unity IDE and VuforiaSDK (It is an Augmented Reality Software Developments Kit (SDK) for mobile devices that enables the creation of Augmented Reality applications).

Videos

Conversations with personal AI Assistant

Controlling appliances using personal AI Assistant

Controlling appliances using Augmented Reality

Digital energy reading and cost of appliances connected to J.A.R.V.I.S.

Useful links:

1. Augmented Reality App Development : https://library.vuforia.com/articles/Solution/Compiling-a-Simple-Unity-Project

2. Android App Development : https://developer.android.com/training/index.html

3. ChatBot using neural network : http://www.wildml.com/2016/04/deep-learning-for-chatbots-part-1-introduction/

Schematics

J.A.R.V.I.S. Android App

Kindly note: Active Internet Connection on phone is required at all times
Instructions :
1. Install the app on an android phone
2. After setting up the hardware and running it successfully, start the app
3. For using Virtual Assistant (Voice), say the keyword, 'Listen Up JARVIS'. This should activate the agent
3.1 Speak your command clearly
3.2 The agent should provide a response
4. To start the augmented reality app, click on the augment button; a blank screen should appear
4.1 Open the Jarvis Augmented App on a different phone (NOTE : The J.A.R.V.I.S. android app should be on a different phone)
4.2 Proceed to instructions given in JarvisAugmented for implementing augmented reality

NOTE : 1. If app is stuck at any point, try restarting the app
2. If a message similar to, 'Could not open Microphone' appears on the screen, restart the app
3. This app is implemented using HC05 Bluetooth module to increase the scope of the project. Hence, HC05 Bluetooth Module is recommended

Code

#include <LiquidCrystal.h>
LiquidCrystal lcd(7,6,5,4,3,2);

int led=13; // declering LED in 13th pin
#define pulsein 8 //defining the pulse input pin 8
unsigned int pusle_count=0; //Declaring Pulse count type as integer
float units=0; //Declaring units as Float
unsigned int rupees=0; //declaring rupees types as unsigned integer

unsigned int temp=0,i=0; //declaring temp& i as unsigned integer

char str[70],flag1=0,flag2=0; // character length as string and declaring flags


void setup() 
{
  lcd.begin(16,2);  // initializing LCD type as 16x2
  Serial.begin(9600); // initializing serial baud rate as 9600
  pinMode(led, OUTPUT);
  pinMode(pulsein, INPUT);
   digitalWrite(pulsein, HIGH);
  lcd.setCursor(0,0);
  lcd.print("Wireless Energy");
  lcd.setCursor(0,1);
  lcd.print("      Meter    ");
  delay(2000);
  lcd.clear();
  lcd.print("By   ");
  delay(2000);
  lcd.clear();
  lcd.print("GSM Initilizing...");
  gsm_init();
  lcd.clear();
  lcd.print("System Ready");
  Serial.println("AT+CNMI=2,2,0,0,0");
  init_sms();
  send_data("System Ready");
  send_sms();
  delay(1000);
  digitalWrite(led, LOW);
  lcd.clear();
}

void loop()    
{
    serialEvent();  // checking serial event
    units=rupees/5.0;  // calculation of unit
    lcd.setCursor(0,0);
    lcd.print("Units:");
    lcd.print(units);
    lcd.print("      ");
    lcd.setCursor(0,1);
    lcd.print("Rupees:");
    lcd.print(rupees);
    lcd.print("      ");
    read_pulse();
    
    if(temp==1)
    {
          check();
    }
}

void serialEvent() // serial event for receiving and checking the incoming sms
{
  while(Serial.available()) 
  {
    if(Serial.find("#"))
    {
      digitalWrite(led, HIGH);
      delay(1000);
      digitalWrite(led, LOW);
      while (Serial.available()) 
      {
      char inChar=Serial.read();
      str[i++]=inChar;
      if(inChar=='*')
      {
        temp=1;
        return;
      } 
      } 
    }
   }
 }

void init_sms() //initializing & configuration of sending message
{
   Serial.println("AT+CMGF=1");
   delay(200);
   Serial.println("AT+CMGS=\"INSERT MOBILE NUMBER HERE\"");
   delay(200);
}

void send_data(String message)
{
  Serial.println(message);
  delay(200);
}

void send_sms()
{
  Serial.write(26);
}

//Reading Pulse from Energy Meter
void read_pulse()
{
    if(!digitalRead(pulsein))
    {
      digitalWrite(led, HIGH);
      units++;
      rupees=units*5;
      while(!digitalRead(pulsein));
      digitalWrite(led,LOW);
    
    }
}

//Checking the status of Message 
void check()
{
     
     if(!(strncmp(str,"units",5)))
      {
        Serial.println("AT+CMGF=1");
   delay(200);
   Serial.println("AT+CMGS=\"INSERT MOBILE NUMBER HERE\"");
   delay(200);
       lcd.setCursor(0,1);
       send_data("Energy Meter Unit:");
       Serial.println(units);
       send_data("\nRupees:");
       Serial.println(rupees);
       delay(200);
       Serial.write(26);
       flag1=1;
       flag2=1;
       temp=0;
      }
     }



void message_sent()
{
  lcd.clear();
  lcd.print("Message Sent.");
  delay(1000);
}

void gsm_init()
{
  lcd.clear();
  lcd.print("Finding Module..");
  boolean at_flag=1;
  while(at_flag)
  {
    Serial.println("AT");
    while(Serial.available()>0)
    {
      if(Serial.find("OK"))
      at_flag=0;
    }
    delay(1000);
  }

  lcd.clear();
  lcd.print("Module Connected..");
  delay(1000);
  lcd.clear();
  lcd.print("Disabling ECHO");
  boolean echo_flag=1;
  while(echo_flag)
  {
    Serial.println("ATE0");
    while(Serial.available()>0)
    {
      if(Serial.find("OK"))
      echo_flag=0;
    }
    delay(1000);
  }

  lcd.clear();
  lcd.print("Echo OFF");
  delay(1000);
  lcd.clear();
  lcd.print("Finding Network..");
  boolean net_flag=1;
  while(net_flag)
  {
    Serial.println("AT+CPIN?");
    while(Serial.available()>0)
    {
      if(Serial.find("+CPIN: READY"))
      net_flag=0;
    }
    delay(1000);
  }
  lcd.clear();
  lcd.print("Network Found..");
  delay(1000);
  lcd.clear();
}

# define CH3 7
# define CH4 4
# define CH5 5
# define CH6 6

 void setup()  
 {  
  Serial.begin(9600);  
  pinMode(CH3, OUTPUT);
  pinMode(CH4, OUTPUT);
  pinMode(CH5, OUTPUT);
  pinMode(CH6, OUTPUT);
   //Provide power to both LEDs
   
   
   //Turn OFF any power to the Relay channels
   digitalWrite(CH3, HIGH); 
   digitalWrite(CH4, HIGH); 
   digitalWrite(CH5, HIGH); 
   digitalWrite(CH6, HIGH); 
   delay(2000); //Wait 2 seconds before starting sequence
  
 }  
 void loop()  
 {  
       
       
   delay(100);
   
   if(Serial.available())
   {
     
  //Serial.println(sum1);
    
    char c = Serial.read();  
    Serial.print(c);
   if(c=='b')
    {
      //coffee off
    }
   else if(c=='c')//light on
    {digitalWrite(CH4  ,LOW);
       delay(100);}
   else if(c=='d')//fan on
    {digitalWrite(CH6,LOW);
       delay(100);}
   else if(c=='k')//coffee on
    {digitalWrite(CH5,LOW);
       delay(100);}
   else if(c=='g')//coffee off
    {digitalWrite(CH5,HIGH);
       delay(100);}
   else if(c=='e')//light off
    {digitalWrite(CH4,HIGH);
       delay(100);}
     else if(c=='f')//fan off
    {digitalWrite(CH6,HIGH);
       delay(100);}
     else if(c=='z')//goodbye
    {digitalWrite(CH6,HIGH);
     digitalWrite(CH4,HIGH);
     digitalWrite(CH3,HIGH); 
     digitalWrite(CH5,HIGH);
     delay(500);
     digitalWrite(CH5,LOW);
     delay(500);
     digitalWrite(CH5,HIGH);
     delay(500);
     digitalWrite(CH5,LOW);
     delay(500);
     digitalWrite(CH5,HIGH);      
     delay(10000);
     }  
    
  }  
 }

J.A.R.V.I.S. : A Virtual Home Assistant