Amazon Echo VR Controller

Virtual Reality is cool. You can travel to places without the hassle of packing your bags, sleeping in a hotel with bed bugs, airport delays, missing your flight. It can take you to virtual worlds that never exist - In the comfort of your pajamas.

With this new technology, we need to reimagine how we interact with Virtual Reality. It's the interaction that makes it real - Visual, sights, sounds, ambiance. The smell, the wind, all our sensory perception.

That's the reason why I started to look into this. What are the next era of interacting with Virtual Reality. And as makers, we are part of this process. We are the bridge. We can take Virtual Reality to the next level.

This is part of a 6 series project exploring Virtual Reality as a Maker. We're not going to create games we are going to explore different ways to create new experiences and interact in VR.

Part 1 - Voice Interaction. Amazon Echo is an amazing household appliance. It has replaced the music, radio player at my house. I got it connected to my home automation. It can control TV, Lights, Sounds. I can ask for weather, traffic information and current news. How about Virtual Reality? We can definitely use it to control my Virtual Reality experience too.

Part 2: Gesture Controller using Particle Photon and Sparkfun RGB and Gesture Sensor APDS-9960

https://www.hackster.io/18937/augmented-reality-virtual-reality-dinosaur-experience-be9c8c

Part 3: Virtual Reality Fireworks - Maker Edition

https://www.hackster.io/RONDAGDAG/virtual-reality-fireworks-using-color-sensor-779ea7

Part 4: ConstructAR - The Holographic Tool Belt

https://www.hackster.io/team-constructar/constructar-the-holographic-tool-belt-b44698

Part 5: Control your "Earth Rover" in Virtual Reality

https://www.hackster.io/RONDAGDAG/control-your-earth-rover-in-virtual-reality-15a9fe

Part 6: Posture Recognition using Kinect, Azure IoT, ML and WebVR

https://www.hackster.io/RONDAGDAG/posture-recognition-using-kinect-azure-iot-ml-and-webvr-e9c4f7

If these projects made you interested in developing Virtual Reality apps, or NodeJS websocket sites in Azure, or creating an Amazon Echo custom skill app; Please click respect and follow me. Feel free to contact me if you have questions. Thanks.

This project has 3 parts:

1. Amazon Skill - This is the app to receive commands from our user.

2. Azure Website - NodeJS chat server - this is the backbone of communication. (Cheapest way)

3. Unity3d Virtual Reality App - A simple Virtual Reality Town to explore.

Data Flow

1. The NodeJS chat server in Azure is the communication piece. It relays commands from Amazon Echo to the Virtual Reality App. 

2. When Alexa hears the keyword, she would contact the NodeJS chat server and login as "Alexa". Then she would broadcast to all members the keyword - in this case "gym". 

3. The Unity3d Virtual Reality App would login as "VR" in the NodeJS chat server in Azure. It would then listen to user "Alexa" for her commands. 

It's a form of a publisher-subscriber pattern using cloud services. The chat server is the backbone of communication. Since there is no direct connection between Alexa and VR app, I just made a free Azure NodeJS Website to handle communication between the two. Azure allows you to have free Azure website.

It can be scalable too. If you decide to expand and create your own multi-player game or even have group Virtual Reality tours where multiple VR applications, logging into a channel, then listening to each other commands.

Amazon Skill

I followed this instructions to create a new skill

https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/developing-an-alexa-skill-as-a-lambda-function

But I have to use a websocket client library so I have to create a NodeJS app locally, zip up all the contents and then upload the zip file instead.

I used the Socket.IO client to communicate from the AWS Lambda to the Azure NodeJS chat server.

https://www.npmjs.com/package/socket.io-client

Here's the intent

{
"intents": [
{
"intent": "TransportIntent",
"slots": [
{
"name": "Location",
"type": "LIST_OF_PLACES"
}
]
}
]
}

LIST_OF_PLACES is defined as barber shop | hardware store | coffee shop | gym

Utterances are

TransportIntent take me to {Location}
TransportIntent take me to the {Location}
TransportIntent transport to {Location}
TransportIntent move to {Location}

The Lambda receives the transport intent with location slot

function onIntent(intentRequest, session, callback) {
console.log("onIntent requestId=" + intentRequest.requestId +
", sessionId=" + session.sessionId);
var intent = intentRequest.intent,
intentName = intentRequest.intent.name;
// Dispatch to your skill's intent handlers
if ("TransportIntent" === intentName) {
handleTransport(intent, session, callback);
} else if ("AMAZON.HelpIntent" === intentName) {
getWelcomeResponse(callback);
} else if ("AMAZON.StopIntent" === intentName || "AMAZON.CancelIntent" === intentName) {
handleSessionEndRequest(callback);
} else {
throw "Invalid intent";
}
}

I call handleTransport function that would connect to the NodeJS chat app. It would login as "Alexa", send a new message of the "location", the it would send the "disconnect" command after the chat server confirmed that it got my message.

function handleTransport(intent, session, callback) {
var repromptText = null;
var sessionAttributes = {};
var shouldEndSession = false;
var speechOutput = "Ok";
var locationSlot = intent.slots.Location;
if (locationSlot) {
var location = locationSlot.value;
socket.on('connect', function(){
  console.log('connected');
  socket.on('login', function(data){
      console.log('login');
      socket.emit('new message', location); //send location as a message
  });
  socket.on('got message', function() {
      socket.disconnect();
      callback(sessionAttributes,
          buildSpeechletResponse(intent.name, speechOutput, 
              repromptText, shouldEndSession));
  });
  socket.emit('add user', 'alexa'); //login as "Alexa`"
});
} else {
speechOutput = "I'm not sure where to go. Please try again";
repromptText = "I'm not sure where to go, take me to the gym";
}
}

NodeJS chat server deployed in Azure

I followed the same structure based from tutorials. It's really just the chat app from socket.io examples. 

https://azure.microsoft.com/en-us/documentation/articles/cloud-services-nodejs-chat-app-socketio/

I tried it locally first. Create a sample nodeJS chat app locally. 

I followed the instructions here.

https://azure.microsoft.com/en-us/documentation/articles/web-sites-nodejs-chat-app-socketio/

Very important:

- you have to create web.config .. 

like this https://github.com/rondagdag/Alexa-VRController/blob/master/web.config

- since it's nodeJS, you have to change this in the web.config

Here's a video to deploy your nodeJS website in Azure. Scott Hanselman has some info too.

http://www.hanselman.com/blog/EnablingWebsocketsForSocketioNodeAppsOnMicrosoftAzure.aspx

If you want to try out the chat site, go here.

https://alexa-vrcontroller.azurewebsites.net/

It's just a chat site. This is how I test my Virtual Reality App. I would login as "Alexa" on the chat site. Send commands there.

Here's the code for the azure website.

https://github.com/rondagdag/Alexa-VRController

I made a few changes to the chat app. I wasn't getting the data properly from AWS lambda, I found out since it's a microservice, i need to acknowledge every message it sends so it would end the transaction properly.

socket.on('new message', function (data) {
// we tell the client to execute 'new message'
socket.emit('got message');
socket.broadcast.emit('new message', {
username: socket.username,
message: data
});
});

so, I added socket.emit('got message') to the chat server.

Then on the AWS lambda side, when I receive the 'got message' acknowledgement, that's when I pass data to Amazon Echo. This will ensure that the transaction ends properly.

Unity3d Virtual Reality App

I started with sample app from this Simple Town - cartoon asset. The scene is SimpleTown_DemoScene. I bought the asset, it's $10.

https://www.assetstore.unity3d.com/en/#!/content/43500

I converted it to a Virtual Reality App using these instructions.

Here's how to setup Gear VR in Unity. A lot of tutorials out there is outdated, here's the easiest way to do it.

These instructions also apply for Oculus Rift. The code works the same, only difference is that you're deploying as exe file rather than apk file to your Samsung Phone.

In order to communicate with the NodeJS chat server, I used socket.io for unity from Asset Store. It's free and easy to use.

https://www.assetstore.unity3d.com/en/#!/content/21721

There's a test scene to learn how to use it.

I added an Empty Object in Unity. Add Socket.IO Component. I replaced to url, instead of using 127.0.0.1:3000, I used this

ws://alexa-vrcontroller.azurewebsites.net/socket.io/?EIO=4&transport=websocket

It will point to my chat server and starts connecting with it. Make sure you setup the chat server to use web sockets. It will not connect if web sockets is disabled. 

I added network.cs script

The script is simple. When I got connected to my chat server, I login as user "VR".

public void OnConnected(SocketIOEvent e)
	{
		Debug.Log("[SocketIO] Open received: " + e.name + " " + e.data);
		var j = new JSONObject(JSONObject.Type.STRING);
		j.str = "VR";
		socket.Emit ("add user",j);
	}

Once I am a registered user to chat server, I can receive all chat messages. If I received a message from user "Alexa", I would just change my player location in Virtual Reality.

public void OnNewMessage(SocketIOEvent e)
{
Debug.Log("[SocketIO] New Message received: " + e.name + " " + e.data);
var username = e.data["username"].str;
Debug.Log (username);
if (username == "alexa") {
message = e.data["message"].str.ToLower();
Debug.Log ("Message:" + message);
switch (message) {
case "barber shop":
  player.transform.position = new Vector3 (-37.8199f, 5f, 50.13747f);
  break;
case "hardware store":
  player.transform.position = new Vector3 (39.82712f, 5f, 71.91634f);
  break;
case "coffee shop":
  player.transform.position = new Vector3 (22.9177f, 5f, 29.13976f);
  break;
case "gym":
  player.transform.position = new Vector3 (20.57586f, 5f, 14.77492f);
  break;
}
}
}

Here's the source for the network.cs, I also attached the apk file.

https://github.com/rondagdag/Alexa-VRController-AmazonEcho/tree/master/unity3d-app

If this project made you interested in developing Virtual Reality apps, creating NodeJS websocket website in Azure, or creating an Amazon Echo custom skill app, Please click "Respect Project" and follow me. Thanks.

Now that this project is done. I really need to visit the gym IRL!