Microsoft Launches HoloLens 2 "Research Mode," Unlocking All Sensors for Computer Vision Projects

Microsoft is positioning its HoloLens 2 mixed reality headset system as a platform for computer vision research, through the inclusion of an improved "Research Mode" which unlocks access to all cameras and sensors.

"Since its launch in November 2019, Microsoft HoloLens 2 has helped enterprises in manufacturing, construction, healthcare, and retail onboard employees more quickly, complete tasks faster, and greatly reduce errors and waste," Microsoft's Marc Pollefeys claims of the platform, which integrates a Windows PC, augmented reality display, and multiple sensors into a single wearable platform. "It sets the high-water mark for intelligent edge devices by leveraging a multitude of sensors and a dedicated ASIC (Application-Specific Integrated Circuit) to allow multiple real-time computer vision workloads to run continuously."

"In Research Mode, HoloLens 2 is also a potent computer vision research device. Compared to the previous edition, Research Mode for HoloLens 2 has the following main advantages: In addition to sensors exposed in HoloLens 1 Research Mode, we now also provide IMU sensor access (these include an accelerometer, gyroscope, and magnetometer); HoloLens 2 provides new capabilities that can be used in conjunction with Research Mode. Specifically, articulated hand-tracking and eye-tracking which can be accessed through APIs while using research mode, allowing for a richer set of experiments."

The Research Mode switch unlocks access to all HoloLens 2 sensors, providing both video and audio streams as well as the ability to access onboard algorithm outputs from simultaneous localization and mapping (SLAM) and 3D mesh generation systems. As well as streaming data from the color camera, which is generally available to applications, Research Mode also provides raw access to the output of the headset's depth camera, the four grayscale cameras used for head tracking, and the headset's inertial measurement unit (IMU).

"The sensors’ streams can either be processed or stored on device or transferred wirelessly to another PC or to the cloud for more computationally demanding tasks," Pollefeys explains. "This opens a wide range of new computer vision applications for HoloLens 2. HoloLens 2 is particularly well suited as a platform for egocentric vision research as it can be used to analyze the world from the perspective of a user wearing the device. For these applications, HoloLens devices’ abilities to visualize results of the algorithms in the 3D world in front of the user can be a key advantage. HoloLens sensing capabilities can also be very valuable for robotics where these can, for example, enable a robot to navigate its environment."

Research Mode is available on the HoloLens 2 now via the Windows Insider beta program, with general availability scheduled for "soon" in an update to Windows 10 for HoloLens. Microsoft has also released a series of example projects for HoloLens 2's Research Mode API on GitHub, under the permissive MIT License; additional documentation is available on the company's developer portal.