Prerequisites
Steps to Set up Arduino IDE before programming the NodeMCU
Steps to Set up Blynk App
Programming the NodeMCU Development Board
C# Script for Unity3D
Steps to configure Unity3D

Published August 21, 2021

Augmented Reality with Internet of Things (IoT)

Control AC Appliances using Augmented Reality Buttons over the Internet from anywhere in the world.

BeginnerFull instructions provided6 hours2,693

Augmented Reality with Internet of Things (IoT)

Things used in this project

Hardware components

NodeMCU ESP8266 Breakout Board

LED (generic)

Jumper wires (generic)

Software apps and online services

Unity

Vuforia

Microsoft VS Code

Arduino IDE

Story

This is a demonstration of Augment Reality (AR) with the Internet of Things (IoT). The AC appliance can be controlled over the internet from anywhere in the world via virtual buttons in Augmented Reality. In this demonstration, I have used an LED but a relay module can be used in place of the LED to control any AC appliance. As soon as the camera recognises the target image, two virtual buttons 'ON' and 'OFF' appear in augmented reality. The appliance or the LED in this case can be controlled by clicking those buttons in real-time in space in the air.

Prerequisites:

Softwares:

Unity Engine: Download - Unity (unity3d.com)
Vuforia Engine: Vuforia Developer Portal
Arduino IDE: Software | Arduino
Visual Studio Code: Download Visual Studio Code - Mac, Linux, Windows

Hardware Components:

NodeMCU Development Board: Nodemcu (ESP8266)
Jumper Wires: Jumper Wire (Male to Male, Male to Female, Female to Female)
LED: LED (5mm, White, 100 Pieces)

Steps to Set up Arduino IDE before programming the NodeMCU:

Select your Operating System properly and install the latest version of Arduino IDE. https://www.arduino.cc/en/software
Install the current Arduino IDE at the 1.8.15 level or later. The current version is on the Arduino website.
Start Arduino and open the Preferences window.
Enter https://arduino.esp8266.com/stable/package_esp8266com_index.json into the File>Preferences>Additional Boards Manager URLs field of the Arduino IDE. You can add multiple URLs, separating them with commas.
In the Arduino IDE Go to Tools -> Board (…) -> Boards manager…
Search for “ESP8266”, click the suggested result and press install.
Install the new board manager for the ESP8266 chip
4. Confirm the installation and choose the NodeMCU board for the current board
Go to Tools -> Board (…) and select “NodeMCU 1.0 (ESP 12-E module)” from the list.

Steps to Set up Blynk App

Download the Blynk App on your phone using this link: https://play.google.com/store/apps/details?id=cc.blynk
Create a new account using your Gmail id.
Login using the same credentials.
Click on ‘New Project’ to create a new project.
Enter the name of your choice, select the board as ‘NodeMCU’, connection type ‘WiFi’and theme as per your choice.
You will receive an authentication token (Like this: XXXXXXXXXXX_y2W68cvU5kG8osz3Kn2H) on your email id. Keep it saved.
Click on the ‘+’ icon again and add a button.
Click on it and set the name of your choice, set the pin to Virtual-V1, toggle set to ‘push’, and rest of the setting as per your choice.
Click the play button at the right top of the application and the configuration is completed.

Programming the NodeMCU Development Board:

Open the Arduino IDE which is preconfigured for the NodeMCU development board.
Paste the code given below:

#define BLYNK_PRINT Serial
#define relay D0 //setting the digital pin as D0
#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <BlynkSimpleEsp8266.h>
char auth[] = "your_authentication_token";
char ssid[] = "wifi_name";
char pass[] = "wifi_password";        //leave blank in case of an open network.
BLYNK_WRITE(V1) //enter the virtual pin that was selected by you previously
{

  int pinValue = param.asInt(); // assigning incoming value from pin V1 to a variable
  digitalWrite(relay,pinValue);

    if(pinValue==1)
  {
    Serial.println("ON");
  }

  if(pinValue==0)
  {
    Serial.println("OFF");
  }

}
void setup()
{
  Serial.begin(9600);
  Blynk.begin(auth, ssid, pass);
  pinMode(relay, OUTPUT);
}
void loop()
{
  Blynk.run();
}

C# Script for Unity3D:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Vuforia;
using UnityEngine.Networking;
public class ARwithIoT_Script : MonoBehaviour
{
public VirtualButtonBehaviour VB_ON;
public VirtualButtonBehaviour VB_OFF;
public string URL_ON;
public string URL_OFF;
IEnumerator GetRequest(string uri)
{
using (UnityWebRequest webRequest = UnityWebRequest.Get(uri))
{
// Request and wait for the desired page.
yield return webRequest.SendWebRequest();
}
}
void Start()
{
VB_ON.RegisterOnButtonPressed(OnButtonPressed_on);
VB_OFF.RegisterOnButtonPressed(OnButtonPressed_off);
}
public void OnButtonPressed_on(VirtualButtonBehaviour VB_ON)
{
StartCoroutine(GetRequest(URL_ON));
Debug.Log("LED IS ON");
}
public void OnButtonPressed_off(VirtualButtonBehaviour VB_OFF)
{
StartCoroutine(GetRequest(URL_OFF));
Debug.Log("LED IS OFF");
}
}

Steps to configure Unity3D:

1. Open Unity Hub.

2. Click on New and Name the project.

3. Add vuforia engine package inside the Unity3D engine to enable unity3D with the AR camera.

4. Go to developer.vuforia.com and create and login into your account. (Example: using Gmail ID)

5. Click and download the file:

6. Click and open the file and click Import to import all vuforia packages. Click ‘update’ if prompted.

7. Delete the main camera from the panel.

8. Right-click>vuforia engine>AR Camera

9. For Target Image:

9.1. Go to the developer.vuforia.com same website and click on Develop.

9.2. Click on Target Manager> Add database.

9.3. Name it as per your choice and select ‘Device’.

9.4. Click on the name> Add Target Image> Browse the target image in your system> choose. Width = 1. Name as per your choice.

9.5. Download this database by selecting the database and download for Unity Editor and click on download.

9.6. Open the file to directly import it into your unity project.

10. To add licence:

10.1. Click on AR Camera and then on the right-hand side find the ‘open vuforia configuration’ and add the licence.

11. To generate licence key:

11.1. Go to the developer.vuforia.com same website and click on Develop.

11.2. Click on licence manager and click on get development key. Give it a name and click confirm.

11.3. Open the licence and copy the entire licence key and paste it in unity editor.

12. Right-click on AR Camera> vuforia engine> image target

13. On the right-hand side, under ‘image target behaviour’, select type as ‘from database’ select the name of the database and name of the image target.

14. Click on image target, go to advanced under ‘image target behaviour’ click on add virtual button.

15. Add one more virtual button by repeating step 10.

16. Rearrange the buttons, one on left and one on the right side of the image target.

17. Rename the virtual buttons as ‘ON’ and ‘OFF’.

18. Right-click on ON and under 3D object click on ‘Plane’.

19. Resize the plane to match the size of the virtual button.

20. Repeat steps 18 and 19 for the OFF button.

21. Add images of ON and OFF in the unity project.

22. Drag and drop the ‘on image’ on the ON button.

23. Drag and drop the ‘off image’ on the OFF button.

24. Change the orientation by changing the Y-axis to 180.

25. Add the C# file in the unity project.

26. Drag and drop the script on the ‘Image target’.

27. Click on Image Target and provide all the data under the ‘Script’ panel.

28. Add the virtual buttons as well as the URLs.

29. Test the application by clicking the play button in the unity editor.

Video Tutorial:

Video recorded by Manipal University Jaipur in which I conducted a two day workshop with my Co-speakers

Code

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Vuforia;
using UnityEngine.Networking;

public class ARwithIoT_Script : MonoBehaviour
{
    public VirtualButtonBehaviour VB_ON;
    public VirtualButtonBehaviour VB_OFF;
    public string URL_ON;
    public string URL_OFF;

    IEnumerator GetRequest(string uri)
    {
        using (UnityWebRequest webRequest = UnityWebRequest.Get(uri))
        {
            // Request and wait for the desired page.
            yield return webRequest.SendWebRequest();

        }
    }

    void Start()
    {
        VB_ON.RegisterOnButtonPressed(OnButtonPressed_on);

        VB_OFF.RegisterOnButtonPressed(OnButtonPressed_off);
       
    }


    public void OnButtonPressed_on(VirtualButtonBehaviour VB_ON)
    {
        StartCoroutine(GetRequest(URL_ON));
        Debug.Log("LED IS ON");
    }

    public void OnButtonPressed_off(VirtualButtonBehaviour VB_OFF)
    {
        StartCoroutine(GetRequest(URL_OFF));
        Debug.Log("LED IS OFF");
    }

}

#define BLYNK_PRINT Serial
#define relay D0
#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <BlynkSimpleEsp8266.h>
char auth[] = "your_authentication_token";
char ssid[] = "your_ssid_name";
char pass[] = "your_ssid_password";
BLYNK_WRITE(V1)
{
  int pinValue = param.asInt(); // assigning incoming value from pin V1 to a variable
  digitalWrite(relay,pinValue);
    if(pinValue==1)
  {
    Serial.println("ON");
  }
  if(pinValue==0)
  {
    Serial.println("OFF");
  }
}
void setup()
{
  Serial.begin(9600);
  Blynk.begin(auth, ssid, pass);
  pinMode(relay, OUTPUT);
}

void loop()
{
  Blynk.run();
}