A Very Touchy Subject

Tapping and swiping are a very intuitive means for interaction with electronic devices, and as such, have become a very popular mechanism for capturing user input in recent years. Extending touch sensing beyond the confines of a touchscreen means additional instrumentation is required. Whether electrical, optical, or acoustic sensing is employed, it calls for additional setup effort and cost.

A group based at The University of Auckland, New Zealand have developed a method they call ActualTouch that enables touch gestures to be collected from a large number of unmodified, everyday surfaces. The device itself is a tiny inertial measurement unit (IMU) sensor attached to a fingertip. Specifically, they redesigned a SparkFun MPU9250 IMU breakout board to achieve a smaller (7mm x 9mm) form factor.

The basic theory of operation resulted from the observation that a finger held in the air exhibits microvibrations, and these microvibrations stop when the finger rests on a surface. Similarly, when a finger is moving through the air, it has a tendency to jitter around a bit. When moving across a surface, the finger is more restricted and movement is noticeably smoother. The researchers initially eyeballed some IMU data to confirm that these phenomena were detectable and that the signals appeared to be distinct from one another.

With signals that appeared to be distinct, it seemed like a task that would be well suited to classification via machine learning. The authors collected data from the IMU sensor with an Arduino Due while performing various gestures. This data was used to train a customized variant of a LeNet convolutional neural network model. Training took place on a dedicated workstation with two NVIDIA GTX 1080 Ti GPUs. Testing of the model yielded a very respectable true-positive and true-negative gesture detection rate of 98.6% and 98.5%, respectively.

Perhaps some of the more obvious applications for ActualTouch are in creating virtual touch surfaces for mixed reality and using any surface in the area to act as a mouse for your laptop. The authors also demonstrated some additional very interesting uses — in one case, they used the back of their smartphone to control a cursor with the hand they were holding it with. The other very interesting use demonstrated was an extension of a smartwatch input surface to the hand and arm around it. Check out the video below to see ActualTouch in action.