Published February 23, 2016 © GPL3+

How to Impress Darth Vader

I want to impress Darth Vader with this Photon/Littlebits project. I tried different things, but only way is to use the Force.

IntermediateFull instructions provided4 hours1,811

Things used in this project

Hardware components

Particle Photon

littleBits bend sensor

sparkfun photon imu

glove

usb power bank

littleBits cloudBit

littleBits Gizmos & Gadgets

littlebits wireless transmitter and receiver

glove

9V battery (generic)

Software apps and online services

Vuforia

Littlebits Cloudbit

Particle IDE

Unity

Story

I want to impress Darth Vader. I tried different things, but only way is to use the Force. So I made one! With the Power of Particle Photon. And then I used a glove and added the power of littleBits cloudBit. I created a mobile App using Unity3d.

Then I made a wearable glove with littleBits, when I close my finger, it would send a signal to my phone to move the Darth Vader Model.

Working with Augmented Reality projects are fun and exciting; but adding Particle Photon or littleBits added a new level of Interaction and magic. If you like to see more, please follow me and add respect to this project. Thanks.

Photon

Well, I found out that Sparkfun sells an IMU module to Photon. I can use that to get the rotation, so I can use it know when I tilt the device, I can trigger an augmented reality app app.

https://www.sparkfun.com/products/13629

The tutorial is easy to follow. Thanks Sparkfun.

https://learn.sparkfun.com/tutorials/photon-imu-shield-hookup-guide

I reused about the same code that I deployed in the photon from the Virtual Reality project

https://www.hackster.io/RONDAGDAG/photon-lightsaber-controller-for-vr-ar-18ba34?ref=challenge&ref_id=22&offset=8

I made my phone as Access Point ... then connect Photon to my phone. Then I can now connect a Unity3d app to my Photon.

Here's the code that I used. Make sure you change the IP address of your phone.

https://gist.github.com/rondagdag/7d139d95e5b6ea2ef8e0

I have a USB battery from Motorola that I attached to my photon to make it portable.

1 / 2

Unity3d

I started learning Unity3d this year. This is a good platform for creating games and visualization. Especially in VR and AR.

My background is C# and .NET that's why I chose Unity3d instead of Unreal. I'm hoping I can reuse some .NET libraries for communication. It's now free to use networking stuff in Unity3d with the new version.

I'm new to Unity3d. There are good courses that I took in Pluralsight that helped me grasp the idea and the platform.

What's good about Unity3d is that it has an asset store. Looking at the asset store, there are quite a few libraries that I can use but I need to buy it.

I found this article:

http://forum.unity3d.com/threads/simple-udp-implementation-send-read-via-mono-c.15900/

So I tried the UDP receive since I only want to receive data from the photon.

I connected the Photon to Unity3d via UDP sockets.

I used this tutorial to create an Augmented Reality App.

Thanks to Edgaras Art.

Instead of the Dino, I replaced it with Darth Vader.

I used this Free Darth Vader Model made by Makeamo downloaded from here:

https://sketchfab.com/models/a9536866d4544b78b94da744226eae2d

Here's what I used to receive UDP packets from Photon:

https://gist.github.com/rondagdag/ee821452fad215a3c07c

At the end of the day, I receive the values from Photon, parse the string and pluck out the one I need. The 6th index is the tilt.

client = new UdpClient(port);
while (true)
{
	try
	{
		IPEndPoint anyIP = new IPEndPoint(IPAddress.Any, 0);
		byte[] data = client.Receive(ref anyIP);
		string text = Encoding.UTF8.GetString(data);
				
                // latest UDPpacket
		lastReceivedUDPPacket=text;
		var rot = lastReceivedUDPPacket.Split(',');
		Value = Convert.ToSingle (rot [6]);
	}
	catch (Exception err)
	{
		print(err.ToString());
	}
}

Once I assigned the "Value" variable. I monitor that inside Unity3d.


void OnAnimatorMove()    
{        
    if (anim) {            
        if (Value < 0.2) {  
                //start moving if photon is tilted upward.                
                Debug.Log ("Move");                
                Vector3 newPosition = transform.position;
                newPosition.x -= moveSpeed; // change the position of Darth Vader                
                transform.position = newPosition;            
        } else             
        {                
                transform.position = initialPosition;
        }        
    }

Here's how you can export an Augmented Reality project to an Android Phone

If you like this project, please add respect and follow me. Thank you.

littleBits cloudBit

Create the Glove. I taped 2 bend sensors to a glove and have a splitter bit for power. Use littleBit shoes to stick it to the glove. Connect the splitter bit to battery power. Then connect the 2 bend sensors to the wireless transmitter. As shown in the pictures.

Connect wireless receiver to cloudBit. Make sure cloudBit is setup properly.

I can't just connect the cloudBit to the glove, because I don't have enough power to run the cloudBit. That's why I used a wireless receiver.

I also attached another marker on the glove. So I can have the glowing hand.

Here's an asset in unity3d that connects to WebSockets, that's what I used to connect to the cloudBit

https://www.assetstore.unity3d.com/en/#!/content/8959 - To save some money, I actually compiled the open source websocket-sharp.dll and include it to my project -http://sta.github.io/websocket-sharp/

- You may also need to copy Newtonsoft.Json dlls to your project. I copied it under Plugins folder.

I also attached a C# code snippet in Unity3d on how to connect to cloudBit using WebSockets.

https://gist.github.com/rondagdag/4bff1724cdf624b04db8

Hopefully this will inspire you to start building Augmented Reality Projects with different hardware integration. Follow me if you're interested in Augmented Reality projects and thanks for adding respect this project.

Code

using UnityEngine;
using System.Collections;
using System;
using System.Security.Policy;
using System.Threading;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using WebSocketSharp;
using WebSocketSharp.Net;

public class LittlebitsWebSocket : MonoBehaviour {

	// receiving Thread
	Thread receiveThread;

	public string access_token = "<Access Token Here>";
	public string device_id = "<Device ID Here>";
	private string uri = "wss://api-stream.littlebitscloud.cc/primus/?access_token=";
	public int value;
	WebSocket ws;
	GameObject energyLight;
	EnergyScript energyScript;
	// Use this for initialization
	void Start () {
		uri = uri + access_token; 

		energyLight = GameObject.FindGameObjectWithTag ("Light");
		energyScript = energyLight.GetComponent<EnergyScript> ();
		init ();
	}
	
	// Update is called once per frame
	void Update () {

	}

	private void init()
	{
		receiveThread = new Thread(
			new ThreadStart(ReceiveData));
		receiveThread.IsBackground = true;
		receiveThread.Start();
	}

	private  void ReceiveData()
	{
		ws = new WebSocket (uri);
		{
			// Set the WebSocket events. 

//connect to littlebits and subscribe to a device
			ws.OnOpen += (sender, e) =>
			{
				var data = new JObject();
				data.Add("name","subscribe");
				var args1 = new JObject();
				args1.Add("device_id",device_id);
				data.Add("args",args1);
				
				ws.Send(data.ToString());
			};
			
// when we receive a message from littlebits, we need to parse the data
			ws.OnMessage += (sender, e) =>
			{
				var data = e.Data.Substring(1,e.Data.Length -2);
				var newData = data.Replace("\\", "");
				JObject json = JObject.Parse(newData);
				var payload = json["payload"];
				if (payload != null)
				{
					var newValue = payload["percent"].Value<int>();
					//if (value != newValue)
					{
						energyScript.LocalVelocity_Y = value / 10;
					}
					value = newValue;
					Debug.Log ("Cloudbit:" + value);
				}
			};
			
			ws.OnError += (sender, e) =>
				Debug.Log ("Error >>" + e.Message);

			ws.OnClose += (sender, e) =>
				Debug.Log(String.Format("WebSocket Close ({0})", e.Code));
			
			#if DEBUG
			// To change the logging level.
			ws.Log.Level = LogLevel.Trace;
			
			// To change the wait time for the response to the Ping or Close.
			ws.WaitTime = TimeSpan.FromSeconds(10);
			
			// To emit a WebSocket.OnMessage event when receives a ping.
			ws.EmitOnPing = true;
#endif
		}
		ws.ConnectAsync ();
	}
}

using UnityEngine;
using System.Collections;
using System;
using System.Security.Policy;
using System.Threading;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System.Collections.Generic;
using System.Net.Sockets;
using System.Net;
using System.Text;


public class PhotonWS : MonoBehaviour {

	// receiving Thread
	Thread receiveThread;
	UdpClient client;
	int port = 33333;

	public float Value;

	public GameObject Light;
	string lastReceivedUDPPacket;
	// Use this for initialization
	void Start () {
		init ();
	}

	private List<float> initialSizes = new List<float>();

	void Awake() {
		// Save off all the initial scale values at start.
		ParticleSystem[] particles = Light.GetComponentsInChildren<ParticleSystem>();
		foreach (ParticleSystem particle in particles) {
			initialSizes.Add(particle.startSize);
			ParticleSystemRenderer renderer = particle.GetComponent<ParticleSystemRenderer>();
			if (renderer) {
				initialSizes.Add(renderer.lengthScale);
				initialSizes.Add(renderer.velocityScale);
			}
		}
	}

	public void Update() {
		UpdateScale ();
	}

	public void UpdateScale() {
		// Scale all the particle components based on parent.
		int arrayIndex = 0;
		ParticleSystem[] particles = Light.GetComponentsInChildren<ParticleSystem>();
		foreach (ParticleSystem particle in particles) {
			particle.startSize = initialSizes[arrayIndex++] * (value / 10);
			ParticleSystemRenderer renderer = particle.GetComponent<ParticleSystemRenderer>();
			if (renderer) {
				renderer.lengthScale = initialSizes[arrayIndex++] *
					Light.transform.localScale.magnitude;
				renderer.velocityScale = initialSizes[arrayIndex++] *
					Light.transform.localScale.magnitude;
			}
		}
	}
		
	private void init()
	{

		// ----------------------------
		// Abhören
		// ----------------------------
		// Lokalen Endpunkt definieren (wo Nachrichten empfangen werden).
		// Einen neuen Thread für den Empfang eingehender Nachrichten erstellen.
		receiveThread = new Thread(
			new ThreadStart(ReceiveData));
		receiveThread.IsBackground = true;
		receiveThread.Start();
	}

	private  void ReceiveData()
	{
		client = new UdpClient(port);
		while (true)
		{

			try
			{
				IPEndPoint anyIP = new IPEndPoint(IPAddress.Any, 0);
				byte[] data = client.Receive(ref anyIP);

				string text = Encoding.UTF8.GetString(data);

				//var n = text.Split('A');
				//print(">> " + n[1]);
				// latest UDPpacket
				lastReceivedUDPPacket=text;
				var rot = lastReceivedUDPPacket.Split(',');

				Value = Convert.ToSingle (rot [6]);

			}
			catch (Exception err)
			{
				print(err.ToString());
			}
		}

	}
}

// This #include statement was automatically added by the Particle IDE.
#include "Kalman/Kalman.h"


#include "SparkFunLSM9DS1/SparkFunLSM9DS1.h"
#include "math.h"

unsigned long lastRead= micros();
char myIpAddress[24];


//////////////////////////
// LSM9DS1 Library Init //
//////////////////////////
// Use the LSM9DS1 class to create an object. [imu] can be
// named anything, we'll refer to that throught the sketch.
LSM9DS1 imu;

///////////////////////
// Example I2C Setup //
///////////////////////
// SDO_XM and SDO_G are both pulled high, so our addresses are:
#define LSM9DS1_M   0x1E // Would be 0x1C if SDO_M is LOW
#define LSM9DS1_AG  0x6B // Would be 0x6A if SDO_AG is LOW

////////////////////////////
// Sketch Output Settings //
////////////////////////////
#define PRINT_CALCULATED
//#define PRINT_RAW
#define PRINT_SPEED 10 // 250 ms between prints

// Earth's magnetic field varies by location. Add or subtract 
// a declination to get a more accurate heading. Calculate 
// your's here:
// http://www.ngdc.noaa.gov/geomag-web/#declination
//#define DECLINATION -8.58 // Declination (degrees) in Boulder, CO.

#define DECLINATION -3.42 // Declination (degrees) in Fort Worth.


int i = 0;
// An UDP instance to let us send and receive packets over UDP
UDP Udp;

uint8_t server[] = { 192, 168, 43, 1}; // ip address of my phone

IPAddress IPfromBytes( server );
unsigned int localPort = 33333;

uint32_t timer;

void setup() 
{

  Serial.begin(115200);
  
   Particle.variable("ipAddress", myIpAddress, STRING);
    IPAddress myIp = WiFi.localIP();
    sprintf(myIpAddress, "%d.%d.%d.%d", myIp[0], myIp[1], myIp[2], myIp[3]);
   
   
   
   
  // start the UDP
  Udp.begin(localPort);

  // Before initializing the IMU, there are a few settings
  // we may need to adjust. Use the settings struct to set
  // the device's communication mode and addresses:
  imu.settings.device.commInterface = IMU_MODE_I2C;
  imu.settings.device.mAddress = LSM9DS1_M;
  imu.settings.device.agAddress = LSM9DS1_AG;
  // The above lines will only take effect AFTER calling
  // imu.begin(), which verifies communication with the IMU
  // and turns it on.
  if (!imu.begin())
  {
    Serial.println("Failed to communicate with LSM9DS1.");
    Serial.println("Double-check wiring.");
    Serial.println("Default settings in this sketch will " \
                  "work for an out of the box LSM9DS1 " \
                  "Breakout, but may need to be modified " \
                  "if the board jumpers are.");
    while (1)
      ;
  }
  
  timer = micros();
  
    lastRead = micros();
}

void loop()
{
    
     
    imu.readGyro();
    imu.readAccel();
    imu.readMag();      
 
    printAttitude(imu.ax, imu.ay, imu.az, -imu.my, -imu.mx, imu.mz, imu.gx, imu.gy, imu.gz);
    
  //printGyro();
  
  delay(PRINT_SPEED);
}


void printGyro()
{
  // To read from the gyroscope, you must first call the
  // readGyro() function. When this exits, it'll update the
  // gx, gy, and gz variables with the most current data.
  imu.readGyro();
  

  // Now we can use the gx, gy, and gz variables as we please.
  // Either print them as raw ADC values, or calculated in DPS.
  Serial.print("G: ");
#ifdef PRINT_CALCULATED
  // If you want to print calculated values, you can use the
  // calcGyro helper function to convert a raw ADC value to
  // DPS. Give the function the value that you want to convert.
  Serial.print(imu.calcGyro(imu.gx), 2);
  Serial.print(", ");
  Serial.print(imu.calcGyro(imu.gy), 2);
  Serial.print(", ");
  Serial.print(imu.calcGyro(imu.gz), 2);
  Serial.println(" deg/s");
#elif defined PRINT_RAW
  Serial.print(imu.gx);
  Serial.print(", ");
  Serial.print(imu.gy);
  Serial.print(", ");
  Serial.println(imu.gz);
#endif

 char  ReplyBuffer[] = "acknowledged"; 
    Udp.beginPacket(IPfromBytes, localPort);
     sprintf(ReplyBuffer, "%f,%f,%f", imu.calcGyro(imu.gx),imu.calcGyro(imu.gy), imu.calcGyro(imu.gz));
   Udp.write(ReplyBuffer);
    
   Udp.endPacket();
}

void printAccel()
{
  // To read from the accelerometer, you must first call the
  // readAccel() function. When this exits, it'll update the
  // ax, ay, and az variables with the most current data.
  imu.readAccel();

  // Now we can use the ax, ay, and az variables as we please.
  // Either print them as raw ADC values, or calculated in g's.
  Serial.print("A: ");
#ifdef PRINT_CALCULATED
  // If you want to print calculated values, you can use the
  // calcAccel helper function to convert a raw ADC value to
  // g's. Give the function the value that you want to convert.
  Serial.print(imu.calcAccel(imu.ax), 2);
  Serial.print(", ");
  Serial.print(imu.calcAccel(imu.ay), 2);
  Serial.print(", ");
  Serial.print(imu.calcAccel(imu.az), 2);
  Serial.println(" g");
#elif defined PRINT_RAW 
  Serial.print(imu.ax);
  Serial.print(", ");
  Serial.print(imu.ay);
  Serial.print(", ");
  Serial.println(imu.az);
#endif

}


Kalman kalmanX; // Create the Kalman instances
Kalman kalmanY;
double gyroXangle, gyroYangle; // Angle calculate using the gyro only
double compAngleX, compAngleY; // Calculated angle using a complementary filter
double kalAngleX, kalAngleY; // Calculated angle using a Kalman filter

// Calculate pitch, roll, and heading.
// Pitch/roll calculations take from this app note:
// http://cache.freescale.com/files/sensors/doc/app_note/AN3461.pdf?fpsp=1
// Heading calculations taken from this app note:
// http://www51.honeywell.com/aero/common/documents/myaerospacecatalog-documents/Defense_Brochures-documents/Magnetic__Literature_Application_notes-documents/AN203_Compass_Heading_Using_Magnetometers.pdf
void printAttitude(
float ax, float ay, float az, float mx, float my, float mz, float gx, float gy, float gz)
{
  float roll = atan2(ay, az);
  float pitch = atan2(-ax, sqrt(ay * ay + az * az));

  float heading;
  if (my == 0)
    heading = (mx < 0) ? 180.0 : 0;
  else
    heading = atan2(mx, my);

  heading -= DECLINATION * M_PI / 180;

  if (heading > M_PI) heading -= (2 * M_PI);
  else if (heading < -M_PI) heading += (2 * M_PI);
  else if (heading < 0) heading += 2 * M_PI;


  double dt = (double)(micros() - timer) / 1000000; // Calculate delta time
  timer = micros();

  // Convert everything from radians to degrees:
  heading *= 180.0 / M_PI;
  pitch *= 180.0 / M_PI;
  roll  *= 180.0 / M_PI;

  double gyroXrate = gx / 131.0; // Convert to deg/s
  double gyroYrate = gy / 131.0; // Convert to deg/s

  
  if ((pitch < -90 && kalAngleY > 90) || (pitch > 90 && kalAngleY < -90)) {
    kalmanY.setAngle(pitch);
    compAngleY = pitch;
    kalAngleY = pitch;
    gyroYangle = pitch;
  } else
    kalAngleY = kalmanY.getAngle(pitch, gyroYrate, dt); // Calculate the angle using a Kalman filter

  if (abs(kalAngleY) > 90)
    gyroXrate = -gyroXrate; // Invert rate, so it fits the restriced accelerometer reading
  kalAngleX = kalmanX.getAngle(roll, gyroXrate, dt); // Calculate the angle using a Kalman filter

double accelX = imu.calcAccel(ax);
double accelY = imu.calcAccel(ay);
double accelZ = imu.calcAccel(az);

 
  char  ReplyBuffer[] = "acknowledged"; 
    Udp.beginPacket(IPfromBytes, localPort);
     sprintf(ReplyBuffer, "%.2f,%.2f,%.2f,A,%.2f,%.2f,%.2f", kalAngleX,kalAngleY, heading, accelX,accelY,accelZ);
     //sprintf(ReplyBuffer, "%.2f,%.2f,%.2f", kalAngleX,kalAngleY, heading);
   Udp.write(ReplyBuffer);
    
   Udp.endPacket();
  
  Serial.println(ReplyBuffer);
  
}

Credits

Ron Dagdag

47 projects • 438 followers

Microsoft MVP award / Lead Software Engineer / Augmented Reality. Developer Passionate to learn about Robotics, VR, AR, ML, IOT

How to Impress Darth Vader

Things used in this project

Hardware components

Software apps and online services

Story

Photon

Unity3d

littleBits cloudBit

Schematics

As Shown

Code

LittlebitsWebSocket.cs

PhotonWS.cs

imu-udp.ino

Credits

Ron Dagdag

Comments

Embed the widget on your own site

How to Impress Darth Vader

How to Impress Darth Vader

Things used in this project

Hardware components

Software apps and online services

Story

Photon

Unity3d

littleBits cloudBit

Schematics

As Shown

Code

LittlebitsWebSocket.cs

PhotonWS.cs

imu-udp.ino

Credits

Ron Dagdag

Comments

Related channels and tags