What is it about
Why did I make it?
How does it work
There are three main modules that make up the project
Results

Published October 9, 2016 © GPL3+

Head Tracking for Wireless 3D First Person Vision

A remote controlled vehicle with stereo cameras on a head-tracked pan-tilt unit and wireless 3D first person control via an Oculus Rift.

AdvancedShowcase (no instructions)13,772

Head Tracking for Wireless 3D First Person Vision

Things used in this project

Hardware components

Arduino UNO

SparkFun XBee Explorer USB

SparkFun XBee Shield

Xbee Pro S1

Corona DS238 MG Digital Servos

Software apps and online services

Arduino IDE

Hand tools and fabrication machines

Soldering iron (generic)

3D Printer (generic)

Story

What is it about?

In remote vehicle operation, video feedback provides the operator with real-time spatial and environmental information. Stereoscopic video feedback can improve upon standard single camera systems by providing the operator with details about scale, depth and motion in a rapid and intuitive form. This supplementary information is especially important when operating in unfamiliar or hazardous environments.

Why did I make it?

This project was for a university mechatronic engineering assignment.

The scope of the design was to test the effects on vehicle control when using a 3D visual telepresence system to navigate surroundings. An effective design would deliver heightened interactivity, control, navigation and spatial awareness to the driver, whilst minimizing the negative effects of motion sickness.

How does it work?

A 3D first person vision system has been developed, which captures stereoscopic perspectives with a pair of small cameras and displays them to a user’s corresponding eyes via a head mounted display. The head mounted display simultaneously captures positional and rotational values of the user’s head, using this data to manipulate a pan-tilt unit on which the cameras are mounted. The effect is delivery of a minimal latency 3D perspective of the remote controlled vehicles surroundings, where the field of view can be mechanically adjusted by rotating the head in the desired direction (see first attached video).

Testing the Pan-Tilt unit - Arduino Uno, XBee wireless tx/rx and Oculus Rift DK2

There are three main modules that make up the project:

Video Capture and Communication (blue)

Headtracker Data Communication (green)

Pan-Tilt unit (PTU) with Controller (yellow)

Breakdown of 3 main modules - by colour

An Arduino Uno served as a remote (vehicle mounted) platform for manipulating the PTU, via two digital servos embedded within a 3D printed enclosure. At a minimum the design required at least two Pulse Width Modulated signal outputs, an RS232 UART/USART Serial Controller, and additional I/O pins for additional sensors/actuators (such as dynamic stability). The Arduino was selected over a conventional microcontroller on the basis of:

High level language with pre-compiled libraries,

On-board power regulator, bootloader, etc

Ease of interface to external devices, such as XBee wireless serial devices

Processing speed and memory control not critical to minimize latency, when combined with already substantial video latency.

Results

Various different hypothesis related to motion sickness, control, cognitive memory retention and depth perception were all tested from the prototype design, but when focusing on hardware performance, the Arduino Uno was a simple and reliable platform.

The main target criteria for a successful design was anticipated as;

A wide enough field of view (>100 degrees)

Adequate video resolution

Accurate and responsive tracking

A high refresh rate

Low Latency

From these, the main issue observed was that to do with minimizing video latency, that is the time from moving your head to receiving the corresponding image to your eyes. The cameras, wireless video communication and display methods created a latency up to 133 milliseconds when stationary, with an additional 26 ms generated via the mechanical response of the head-tracking, serial comm's and mechanical adjustment of the PTU. This 26 ms of data processing/signal generation is insignificant when compared to that of the video latency.

In replicating this project, there are many issues regarding video equipment I would reconsider to increase image quality whilst minimizing latency. However, regarding the elements relating to the Arduino, I would consider a smaller packaged board (such as Arduino Pro Mini) and faster operating servo motors as the only limitation experienced was in the rate of PTU adjustment, not in speed of processing and I/O capacity.

Thank you, Arduino, for meeting my needs in what would otherwise be a daunting project.

Arduino Servo Control from XBee serial receive

#include <Servo.h>
Servo yawservo;  // create servo object to control a servo
Servo pitchservo;  // create servo object to control a servo

String inputString = "";         // a string to hold incoming data
boolean yawComplete = false;  // whether the string is complete
boolean pitchComplete = false;  // whether the string is complete

void setup() {
  // initialize serial:
  Serial.begin(9600); // reserve 200 bytes for the inputString:
  inputString.reserve(200);
  yawservo.attach(9);  // attaches the servo on pin 9 to the servo object
  pitchservo.attach(10);  // attaches the servo on pin 9 to the servo object
}

void loop() {
  serialEvent(); //call the function // print the string when a newline arrives:
  if (pitchComplete) {
    pitchservo.write(inputString.toInt());                  // sets the servo position according to the scaled value 
    delay(1); 
    inputString = "";
    pitchComplete = false;
  }

    if (yawComplete) {
    yawservo.write(inputString.toInt());                  // sets the servo position according to the scaled value 
    delay(1); 
    inputString = "";
    yawComplete = false;
  }
}

void serialEvent() {
  while (Serial.available()) {
    // get the new byte:
    char inChar = (char)Serial.read();
    if (inChar == 'x')
    {
      pitchComplete = true;
      break;
    }
        if (inChar == 'y')
    {
      yawComplete = true;
      break;
    }
    inputString += inChar;
  }
}

Credits

twhi2525

1 project • 16 followers

Head Tracking for Wireless 3D First Person Vision