Beyond the Beats
By adding a few more sensors, the headphones we already have could provide more intuitive and immersive interactions with digital devices.
Generally speaking, headphones are designed to do one thing — output audio. Sure, there are some additional functions found fairly frequently, like volume control or noise cancellation, but these functions still all revolve around playing audio. Given that headphones are among the most popular wearable devices on the market, might this narrow scope of focus represent a missed opportunity? Could the existing sensors in commercial headphones be leveraged to provide new types of interactions with our electronic devices? And better yet, what if new types of sensors were added to standard headphones? What use cases might this enable?
It is exactly these types of questions that a team of engineers from Microsoft Research set out to answer. They took a Research through Design approach to the question and hacked away at some headphones to see what new types of applications they could enable, and then determine how they impact the user experience. Broadly speaking, they wanted to enable both intentional inputs that could be used to control devices, as well as background sensing of user activity.
Towards that end, they explored the possibilities associated with the use of sensors like microphones, proximity sensors, motion sensors, inertial measurement units (IMUs), depth cameras, and LiDARs. Some of the ideas that they had for the use of these sensors involved tracking head movements, body postures, and hand gestures. They also wanted to look for opportunities to provide users with more information, and to enable more intuitive and immersive interactions with digital devices.
In one demonstration, headphones equipped with an IMU were paired with a video call application. By detecting when the wearer of the headphones turns their head and shifts their attention away from the screen, perhaps to talk with someone in person, the call is automatically muted and blurred to preserve privacy. A similar setup was used in a case where a person was engaged in more than one video call simultaneously. By tracking their focus, the system could automatically unmute only the call that they are attempting to actively engage with. It is not all business though, there is some time for fun and games as well. This same function could be used to direct speech at a specific individual in a multiplayer video game, for example.
Additional use cases were enabled with the addition of a LiDAR sensor. The team explored the use of common gestures, rather than relying on on-screen or on-headphone buttons. In one case, it was shown how an individual could cup their hand to their ear, in a universal symbol that indicates one is having trouble hearing, to turn up the volume and enable a noise cancellation feature.
Further demonstrations showed how the enhanced headphones could animate an avatar for use in camera-off video calls, or control movements and other actions in video games. The team also thought through a design space for headphone interactions. Through exploring the type of input gesture and the context within which the wearer executes the action they define a framework that can help others explore the many possibilities of this platform.
In the future, the researchers plan to experiment with additional types of sensors, and also with other wearable, and non-wearable devices beyond headphones. They also hope that their work will inspire others to rethink the devices that we use everyday to consider how they might be reimagined.