Kiss Your Keyboard Goodbye

Alternative user interfaces have been increasing in popularity in recent years, however, device control is still primarily focused on methods involving the hands and eyes. For the majority of situations, this is a good option. For most of us, the lion’s share of our interactions with the world occur through visuo-tactile experiences mediated by our visual system and hands, after all. But there are many cases where we would like to interact with a device while our eyes and hands are focused on other tasks, like while driving. Moreover, people with certain disabilities may not have the use of their eyes, hands, or both.

A common solution to this dilemma involves using voice recognition technologies to accept user requests. This can be paired with audibly spoken responses to provide feedback. The rise of the smart speaker proves that this approach has merit, but it is not without its drawbacks. Voice recognition can be inaccurate and impractical when there is a lot of background noise, as is the case in public spaces. Oftentimes, it is also undesirable from a privacy standpoint to say certain things out loud — consider passwords, or other sensitive information, for example.

A trio of researchers at the University of Chicago has recently put forth a proposed solution to this problem. Their university may be known as the place where fun goes to die, but this team is having a lip-smacking good time with their lip-based user interface called LipIO. As the name implies, this interface can accept inputs as well as provide outputs. It was constructed with off-the-shelf hobbyist-grade components, and because of its potential to assist those with disabilities, they have open sourced all of their work.

LipIO consists of a thin, flexible plastic sheet with conductive traces, terminated in five electrodes, laid out on both the top and bottom of the sheet. This sheet is affixed to the skin above the lip, and the electrodes on the bottom come into contact with the lip to provide the system’s outputs via electrostimulation. The electrodes on the top layer receive inputs either from the tongue or lower lip via capacitive touch sensing.

A Seeeduino XIAO development board with a Microchip SAM D21 microcontroller handles the device’s inputs and outputs, and also interactions with external devices. An MPR121 capacitive touch sensor recognizes inputs from the electrodes, while a RehaStim electrical stimulator was included to provide user feedback. Arduino IDE and Adafruit capacitive sensing libraries were used for development. The researchers note that these particular components are not essential, and substitutions can easily be made if you happen to have alternative parts on hand.

The prototype device is fairly large and obtrusive, with wires running from the lip and behind the ears to the control unit, but that could most certainly be greatly improved with some future refinements. LipIO will probably always be relegated to special use cases, however, due to the mechanism of action. Repeatedly licking one’s lips in public is not exactly socially acceptable, and would lead to more than a few misunderstandings. However, there are many use cases for the system that are very compelling.

The team demonstrated how LipIO can, for example, integrate with a navigation app to provide directions to a bicyclist while they are on the go. The device was also demonstrated tuning a guitar while a musician is performing. Many opportunities in virtual reality and gaming also exist. And of course LipIO could also enable people with certain medical conditions to interact with electronic devices in a way that was previously impossible.

At present, the researchers are exploring ways to make LipIO more compact so that it will be more practical for real-world use cases. They are also evaluating the social aspects of the device in an effort to make its use more socially acceptable, perhaps to limit the number of times that they are slapped while running experiments in the wild.