Face Your Emotions

You can tell a lot about a person by the expression on their face. As it turns out, these same visual cues can be deciphered by computers, which presents opportunities for improved human-computer interactions. By tracking facial expressions throughout the day, it is also possible to give one insight into their own mental and physical activities.

To simplify the process of continually tracking facial expressions, a team from Cornell University developed a device they call NeckFace. Two different versions of the device were created and tested: the first is a neckband that is draped around the back of the neck, with a pair of cameras just below shoulder level; the second iteration of the device is a necklace, with the camera housed in a pendant-like casing hanging below the neck. Both devices contain an infrared (IR) camera, a near-IR LED and an IR narrow band-pass filter.

Processing of the IR imagery was performed with a custom deep learning pipeline named NeckNet. NeckNet is based on a ResNet34 model, and it outputs 52 parameters that represent the facial expressions that it interprets as being shown in the IR data.

To validate their design, the researchers recruited a cohort of thirteen individuals to help assess the effectiveness of NeckFace. For comparison purposes, baseline facial movement data was collected using a TrueDepth 3D camera on an iPhone. Participants were directed to sit, walk, and at times remount the NeckFace device. A total of 52 different facial expressions were captured throughout the course of the study.

It was found that NeckFace was able to detect facial movement nearly as well as the baseline measurements. The neckband version of the device was found to outperform the necklace, likely because having a camera on each side of the face provides additional information that is not available to the necklace version.

While NeckFace was found to perform well in the study, it should be noted that some people are not as expressive as others. NeckFace is incapable of collecting meaningful data from people that commonly sport a poker face.

Limitations aside, NeckFace does present a not-to-terribly invasive means of collecting important data. Perhaps with further work the device could be further miniaturized. The team envisions NeckFace being used to give medical doctors better insight into mental health issues, which could help to fine-tune treatments.

If NeckFace has captured your interest, you may also want to check out this device you can build yourself that correlates your emotional states with the web sites you visit, to give you actionable insights about your time spent on the web.