[Tutorial] Augmented Reality with Internet of Things (IoT)

This is a demonstration of Augment Reality with Internet of Things. My home office temperature and humidity is displayed on virtual dashboard which is augmented on top of a real cardboard. There is a virtual button which is augmented on the floor of the cardboard. On tap of the button, dashboard disappears. This technology has lot of potential and I am curious what you build.

Before we dive into technical nitty gritty, watch the demo

I have been asked by many University students how to recreate this project easily. So I am trying to document each steps so that you can get started quickly. If you face issues, let's discuss on the github repo so that everyone can contribute.

Prerequisite

• Unity - I am using version 2018.4.34. If you can, stick to this version as I heard different versions sometime don't work same.
• Vuforia - You can install it through Unity Hub itself. You also need to create a developer account in https://developer.vuforia.com/
• Arduino IDEA - You need to install it and also prepare it to upload ESP8266 sketches. You can follow this tutorial
• Visual Studio Code or your favorite editor
• MQTT client - Download from here

This project has two high level components.

• Sending sensor data to MQTT from Node MCU ( ESP8266 )
• Display sensor data from MQTT on AR dashboard

Send Sensor Data To MQTT

Connect DHT11 sensor to ESP8266 as shown in the image above

"+" to 3V3 Pin

"-" to GND Pin

"Out" to D4 pin

Clone the repo to your computer

git clone https://github.com/just4give/ar-sensor-unity.git

Open SensorMCU.ino file in Arduino IDE update below 3 lines. Put your Wifi SSID, password and choose a topic name such as "yourname/esp8266/sensor"

const char* ssid = "";//put your wifi network name here
const char* password = "";//put your wifi password here
const char* topic = "";

After changing the code, upload the sketch and check logs in serial monitor. You should see temperature and humidity printed every minute.

Now, to make sure sensor data is sent to MQTT, open MQTT.fx client you downloaded before. Then create a new connection profile.

Broker Address = broker.hivemq.com

Broker Port = 1883

Client ID = Click on "Generate" button to get random id

Once profile is created, connect to the broker. Then go to "Subscribe" tab and put your topic name and click on "Subscribe" button. Wait for a minute and you should see data appearing from your ESP8266 board!

Congratulation! You are now successfully sending sensor data to MQTT broker. You are half way through!

Prepare Your Target

Your Augmented dashboard will be attached to a target image. Print target_image.jpeg from my repo which would be your target. You can use any other image of your choice. For that you need to create image target in Vuforia developer account. I will explain that in a bit. For now, just print out the image or open on your phone or tablet.

Create AR Dashboard on Unity

Open Unity Hub, click on "ADD" button and choose ar-sensor-unity folder which you cloned before. You should see a project imported as below.

Click on the project which will open the project in Unity and should look like below.

You will notice, main view is kind of empty. Now, drag "sensorscreen" to "Untitled" (top left corner) and you should see "sensorscreen" added as below image.

Now, right click on "Untitled" and remove the screen. Then open Assets/Scripts/SensorReader.cs file in VS code or any editor and type the topic name you used in ESP8266.

Now, attache an USB camera to your computer and click on play button. Focus your camera on target image and you should see virtual dashboard augmented through camera view on Unity.

Congratulation!!! You have created your AR project successfully.

Your Own Image Target

So far in this project you have used my Vuforia image target. It won't work if I remove my Vuforia key from developer portal. So if you are seeing any issue when reading this article, you need to create your own image target.

Visit https://developer.vuforia.com/ and register your account. Then log into your account and navigate to "License Manager" page. Click on "Get Development Key". In the next page, type a license name, just put your name and click on "Confirm". You should see your license. Click on your license and on next page, copy the license key.

Now, head over to Unity, click on "ARCamera", then click on "Open Vuforia Engine Configuration"

Then paste the license key under "App License Key" field and click on "Add License". This will open Vuforia developer portal. Head over to "Target Manager", then click on "Add Database" button and create one.

Now, click on the database which will take you to below page.

Before you add new image target, first pick one image from internet. I am going to use this.

Now, click on "Add Target" button. Pick the image from your computer.

Type = Single Image

Width = 0.2 scale of target. My target would be 0.25 meter. So 0.25x0.2=0.05

After few minutes, you can see the rating of your image. Higher rating means higher accuracy during augmentation.

Now, select the target and click on "Download Database". Once downloaded, double click the file and import to Unity.

Then inside Unity, select Image Target, database and image as below.

That's it ! Click on play button and focus camera on new target image. I opened the image on iPad and used as my target.