Augmented Reality / Virtual Reality Dinosaur Experience

Mixing Augmented Reality and Virtual Reality opens up a new world of Interaction. By scanning an object, in this case a picture, it overlays a Dinosaur that can be controlled by Gesture sensor connected to Particle Photon.

This is part 2 of a 6 series project exploring Virtual Reality as a Maker. We're not going to create games, we are going to explore different ways to create new experiences and interact in VR. 

Part 1: Amazon Echo VR Controller

https://www.hackster.io/RONDAGDAG/amazon-echo-vr-controller-724c89

Part 2: Gesture Controller using Particle Photon and Sparkfun RGB and Gesture Sensor APDS-9960

Part 3: Virtual Reality Fireworks - Maker Edition

https://www.hackster.io/RONDAGDAG/virtual-reality-fireworks-using-color-sensor-779ea7

Part 4: ConstructAR - The Holographic Tool Belt

https://www.hackster.io/team-constructar/constructar-the-holographic-tool-belt-b44698

Part 5: Control your "Earth Rover" in Virtual Reality

https://www.hackster.io/RONDAGDAG/control-your-earth-rover-in-virtual-reality-15a9fe

Part 6: Posture Recognition using Kinect, Azure IoT, ML and WebVR

https://www.hackster.io/RONDAGDAG/posture-recognition-using-kinect-azure-iot-ml-and-webvr-e9c4f7

If these projects made you interested in developing Augmented Reality / Virtual Reality apps, or tinker with Particle Photon and Sparkfun RGB & Gesture Sensor, or just learned something from this article, please click respect and follow me. Feel free to contact me if you have questions. Thanks.

There are 2 parts on this project:

• Gesture App

• Augmented Reality/Virtual Reality App

WorkFlow:

• The AR/VR app scans for target image and overlays the Dinosaur. It connects to particle photon with Gesture Sensor.

• The user can swipe his/her hands and the Dinosaur will start jumping, rotate or calm down.

• The user can also select the VR button and will be transported to VR Dinosaur world, then vice versa.

Gesture App:

I replicated this project and modified it to communicate via TCP. 

https://www.hackster.io/krvarma/particle-core-gesture-11458a

I also published a Particle Library for SparkFun RGB and Gesture Sensor - APDS-9960 that you can include in your future projects.

Creating a library from an arduino source code for Particle Cloud is tricky.

You have to remove references to Wire.h and Arduino.h. Then Add "application.h". You have to do this on both .h and .cpp files.

//#include 
//#include 
#include "application.h"

https://github.com/rondagdag/apds9960_particlelib

I also found out that TCPClient is part of Particle Library. So I can easily create a TCPServer. I learned that from this code 

https://gist.github.com/dmiddlecamp/737c18b383f56babdfe8

and from this Hackster Project 

https://www.hackster.io/middleca/sending-sound-over-the-internet-f097b4

In which I definitely respect.

This are the variables involved.

TCPClient gestureClient;
TCPClient checkClient;
TCPServer gestureServer = TCPServer(3443);

In my startup, I also publish the IP Address of the device.

Particle.variable("ipAddress", myIpAddress, STRING);
IPAddress myIp = WiFi.localIP();
sprintf(myIpAddress, "%d.%d.%d.%d", myIp[0], myIp[1], myIp[2], myIp[3]);
Serial.println(myIpAddress);
gestureServer.begin(); 

In order to know the ip address, you would do something like this.

https://api.particle.io/v1/devices//ipAddress?access_token=

This is the code to publish the Gesture to particle and also to the TCP Client.

void publishEvent(char* szEvent){
Particle.publish("GESTURE-EVENT", szEvent, 60, PRIVATE);
if (gestureClient.connected()) {
gestureClient.println(szEvent);
}
}

This is how the publish Event is called. When a gesture is detected, it publishes the event.

void handleGesture(){
if(apds.isGestureAvailable()){
char szEvent[8];
szEvent[0] = 0;
switch ( apds.readGesture() ) {
case DIR_UP:            strcpy(szEvent, "UP");      break;
case DIR_DOWN:          strcpy(szEvent, "DOWN");    break;
case DIR_LEFT:          strcpy(szEvent, "LEFT");    break;
case DIR_RIGHT:         strcpy(szEvent, "RIGHT");   break;
case DIR_NEAR:          strcpy(szEvent, "NEAR");    break;
case DIR_FAR:           strcpy(szEvent, "FAR");     break;
}
if(strlen(szEvent) > 0){
Serial.println(szEvent);
publishEvent(szEvent);
}
}
}

Here's the code:

https://gist.github.com/rondagdag/83bde55c4cce92fc1185dc72ad72a0c8

In the end, It looks something like this:

Augmented Reality/Virtual Reality App

I followed the tutorial from Edgaras Art and created an Augmented Reality App.

I want to expand the project and use Gear VR, so I followed the instructions from here.

https://developer.vuforia.com/library/articles/Solution/Integrating-Gear-VR-and-the-AR-VR-Sample-in-Unity-5-3-and-above

I have to download the sample app and integrate it to my dinosaur app

https://developer.vuforia.com/downloads/samples#loginModal

After I got the Sample app working, I added the dinosaur to the ImageTargetStones GameObject.

I also added it to the VR Environment and resized/moved it accordingly.

I tested it. And it worked.

In order to communicate with my Particle Photon, I need to add a way to receive TCP packets. I added PhotonConnection GameObject

Created a new Script. 

https://github.com/rondagdag/ARDinoController/blob/master/Assets/Scripts/socketScript.cs

The attached the Dino to it.

When the TCP receives a response. It would figure out which command and sends it to the Dino Controller.

The SocketScript uses TCP connection

https://github.com/rondagdag/ARDinoController/blob/master/Assets/Scripts/TCPConnection.cs

You have to modify the IpAddress and the port corresponding to your Photon.

	public string conName = "photon";
	//ip/address of the server, 127.0.0.1 is for your own computer
	public string conHost = "192.168.1.36";
	//port for the server, make sure to unblock this in your router firewall if you want to allow external connections
	public int conPort = 3443;
	//a true/false variable for connection status
	public bool socketReady = false;

This is how you would receive the socket data which contains the command. All you have to do then is to set the Action property of the DinoController.

void SocketResponse() {
string serverSays = myTCP.readSocket();
//serverSays = serverSays + "";
if (serverSays != "") {
Debug.Log("[SERVER]" + serverSays);
command = serverSays.Substring(0,serverSays.IndexOf("\r\n"));
switch (command) {
case "FAR":
dinoController.action = DinoController.Action.Hit;
break;
case "NEAR":
dinoController.action = DinoController.Action.Jumping;
break;
case "LEFT":
dinoController.action = DinoController.Action.LookLeft;
break;
case "RIGHT":
dinoController.action = DinoController.Action.LookRight;
break;
case "UP":
dinoController.action = DinoController.Action.Jumping;
break;
case "DOWN":
dinoController.action = DinoController.Action.Idle;
break;
}
}
}

The Dino Controller is attached to the Dinosaur Game Object.

The code looks like this. When it received an Action, it would change the animation.

https://github.com/rondagdag/ARDinoController/blob/master/Assets/Scripts/DinoController.cs

void Update () {
switch (action) {
case Action.Idle:
DoIdle();
break;
case Action.Jumping:
//_animation.CrossFade (jump.name);
switch (Random.Range (0, 3)) {
case 0:
_animation.CrossFade (jump.name);
break;
case 1:
_animation.CrossFade (leftJump.name);
break;
case 2:
_animation.CrossFade (rightJump.name);
break;
}
break;
case Action.LookLeft:
vector.Set(0,transform.rotation.y + 1,0);
transform.Rotate(vector);
DoIdle();
//transform.rotation = Quaternion.LookRotation(vector);
break;
case Action.LookRight:
//_animation.CrossFade (rightJump.name);
vector.Set(0,transform.rotation.y - 1,0);
transform.Rotate(vector);
DoIdle();
//transform.rotation = Quaternion.LookRotation(vector);
break;
}
}

That's about it. 

Observations:

• Programming in Gear VR is easy in Unity especially with the latest version. I am using an older version of Gear VR and it would pause to cool down after running in VR for awhile. Future improvement is to reduce the size of the Dinosaur Model.

• The Gesture sensor is not perfect. Depending on how fast I swiped. And since it's a video feed to Gear VR, coordination with my hand is not easy.

• It's not easy to test, it gets old going back and forth wearing VR headset.

• Just want to thank all the other projects that I found here in Hackster that helped me build this project. 

If this project made you interested in developing Augmented Reality / Virtual Reality apps, or tinker with Particle Photon and Sparkfun RGB & Gesture Sensor, or just learned something from this article, please click "Respect Project" and follow me. Thanks.