Who Was That Masked Man?

Augmented reality (AR) and virtual reality (VR) technologies have improved dramatically over the past few decades, to the point where very impressive capabilities are now available at a reasonable price point. Yet despite these advancements, AR and VR applications are quite limited. It is not exactly a common occurrence to see anyone using these devices in the course of normal, daily interactions with others. That raises the question as to why people are not adopting the technology.

One likely explanation is that wearing the headsets is, well... weird. While some devices have passthrough features that enable the wearer to see their surroundings, no one can see the eyes of the wearer. Eye contact is important in social interactions, and when it is removed, these interactions can feel very unnatural and uncomfortable. Having experienced the strangeness of these interactions firsthand, some engineers from Facebook Reality Labs came up with the idea of building reverse passthrough displays into AR and VR headsets. A reverse passthrough display shows outside observers a rendering of the eyes of the person wearing the headset.

The centerpiece of this enhancement to existing headsets is a 3D display. While a 2D display could be used, it would create a visually jarring and unnatural experience. The 3D display makes the rendered image look far more realistic, and also conveys information about direction, which is crucial for simulating eye contact. A microlens-array light field display was chosen because it is thin, light, and simple to construct. Further, perspective naturally shifts with this type of display — observers in different locations will see the correct perspective for where they are located.

Images of the wearer’s eyes are captured with infrared cameras located inside the headset. Using invisible infrared light allows the headset to capture images without being a distraction from the AR/VR experience.

While not perfect, the result is pretty good. It does not fall into the uncanny valley and make for an unsettling experience, but it is not completely convincing either. In its current state, the reverse passthrough display is a taste of things to come in the future, but not something yet ready for the shelves of your local electronics store.