A New Way to See the World

In recent years, a number of efforts have been undertaken to restore sight to the blind via technological advancements, such as brain-computer interfaces, that bypass damaged visual pathways to provide some form of visual perception. These solutions are not exactly commonplace today, however, and for good reason. While they have shown some promise, we simply do not understand the brain’s visual system well enough to achieve high-fidelity vision restoration. As such, existing systems are of limited real-world value. And that makes the highly-invasive and risky procedures required for their installation too much to stomach for the majority of potential users.

It may be a long time before we fully understand the mysteries of human vision, but a group at the always-creative lab of Pedro Lopes at the University of Chicago has come up with a solution that may help tide us over until we do. Rather than tying sensors into the brain, the team has proposed that visual information be transmitted via a tactile interface affixed to the back of the hand.

This may sound odd at first blush, but imagine for a moment that you are reaching for the handle of a coffee cup. As your hand moves past the handle, the back of your hand can feel right where it is in space. And as your hand gets closer, the intensity of the feeling increases. Similar functionality could be provided when reaching for a handrail on a staircase, or when searching for keys that have been dropped on the floor. This is clearly not like real vision, but it does sound like it might be very helpful for carrying out tasks that normally require vision all the same.

The team’s system is composed of a vision module and a tactile module. The vision module consists of a miniature camera mounted on the palmar side of the user’s hand, allowing it to capture images from the same perspective the hand uses to interact with objects. The camera is very small — just 10×10×5 mm — ensuring that it stays out of the way while still maintaining a 60-degree field of view. The device is designed for real-time interaction, and is capable of transmitting images via USB at a rate of 15 frames per second to a processing unit running a Python-based OpenCV pipeline.

The tactile module uses electrotactile stimulation to provide feedback, allowing the user to feel the shape and position of objects. The electrotactile array consists of 30 flexible electrodes arranged in a 5×6 grid, fabricated using a polyimide-based flexible PCB. Each electrode measures 8mm in diameter and they are spaced 15mm apart — slightly larger than the two-point discrimination threshold of the back of the hand, ensuring clear and distinct sensations. The system employs a multiplexed architecture controlled by an ESP32 microcontroller, which directs electrical signals to specific electrodes based on the processed image data.

The researchers conducted a study involving both blind and sighted (but blindfolded) participants to compare the effectiveness of the hand-based system with a traditional head-mounted sensory substitution device. While both approaches allowed users to interact with objects successfully, the hand-based perspective offered unique benefits. Participants using the hand-mounted system exhibited more natural and ergonomic hand movements, suggesting that this method of sensory substitution could be particularly useful for fine motor tasks. Additionally, when given the option, all participants chose to use both systems together, demonstrating the potential advantages of combining different perspectives.

While restoring sight remains an elusive goal, innovations like this go to show that there are many ways to enhance perception beyond the traditional senses. By rethinking how we process and transmit sensory information, we can create new pathways to experiencing the world — ones that do not require invasive surgery or a deep understanding of the brain’s complexities. And as it turns out, we may have the answer right in the palm of our hands.