Face Interface

Emotions play a fundamental role in how humans perceive, interpret, and respond to the world around them. Accordingly, integrating emotional understanding into interactions with machines can lead to more intuitive, empathetic, and ultimately more effective interactions. Whether it is a chatbot providing customer support, a virtual assistant aiding in productivity, or a robot assisting in healthcare settings, considering human emotions can greatly enhance user experience and outcomes.

By recognizing and responding to human emotions, machines can bridge the gap between artificial intelligence and human psychology, fostering deeper connections and more meaningful interactions. However, accurately capturing human emotions poses significant challenges. Emotions are complex and nuanced, often expressed through a combination of verbal and non-verbal cues such as facial expressions, tone of voice, and body language. Additionally, individuals vary widely in how they express and perceive emotions, making it difficult to develop universal algorithms or models for emotional recognition.

These challenges did not stop a team of researchers at the Ulsan National Institute of Science and Technology in South Korea from trying to build a device that can accurately recognize human emotions. This device takes the form of a flexible and transparent personalized skin-integrated facial interface (PSiFI) that is worn over the face. The onboard sensors can capture both verbal and non-verbal information from the wearer, which is then interpreted by a custom algorithm to infer their emotional state in real-time. To make the system as unobtrusive as possible, it is self-powered and wireless.

Each mask is highly personalized — photos of a user are scanned, then three-dimensional models are created from them. This model is used to create the PSiFI from a soft, stretchable material. Embedded within the mask are strain and vibration sensing units made from PDMS film as a dielectric layer, and a PEDOT:PSS coated PDMS film that serves as electrodes. These sensors were selected to detect facial expressions and also vocal vibrations for speech recognition. A triboelectric nanogenerator was also incorporated into the design to supply power for the device by harvesting energy from normal facial movements.

A small wireless radio transmits sensor data to a receiving station connected to a laptop for analysis. As previously noted, interpreting another individual’s emotional state can be challenging even for other humans. As such, the team designed and trained a convolutional neural network to interpret the sensor data and correlate it with certain emotional states. Experimentation showed that this system was highly accurate. It could correctly classify an individual’s emotional state in better than 93% of cases, on average.

To demonstrate how the device could be utilized in a real world situation, the team built a virtual reality concierge application. The goal of this application was to provide users with certain services, but the services were tailored to the specific emotions the user was experiencing. It was shown that this service could, for example, provide personalized recommendations for music, movies, and books that were appropriate for each person’s emotional state.

While the mask is minimally intrusive, it is still noticeable, and would likely be uncomfortable for long-term use. As such, the PSiFI is probably best suited for niche applications at this time. But if the technology advances, it may become practical for a wide range of use cases in the future.

If you want to dig a bit deeper into how emotions can shape our interactions with technology, you might be interested in checking out this project.