Haptapp Uploader - read text from haptic feedback

What is haptapp?

Haptapp is my attempt to create a better keyboard. Unlike the conventional keyboards, it has only one button. The keyboard layout is a binary tree of letters that can be memorized to type blindly. Here is a quick explanation of how it works:

Pre-history

Haptapp started back in 2007 as a method of hands-free typing designed for EEG devices and BCIs. Currently it projects English alphabet to a 2D circular binary tree that can be memorized by playing the Neuratype game. Each sector of the map has a unique rhythm assigned to it. After about 30 minutes of playing one can learn to easily convert any English text to rhythmical patterns and type blindly on a smartphone with a single finger. The rhythmical patterns can also be used to send text via light, sound, vibrations, touch, electrical spikes or direct neural stimulation.

The initial tests and project presentations revealed the highest interest among visually impaired people. Haptapp does solve a big pain of reading and typing for blind people, but the JavaScript game built in canvas relied fully on vision and wasn't accessible. My next goal was to develop an accessible method of learning to read Haptapp messages that doesn't require eyes or ears, this way it can be learned as a background process and can be used by visually or hearing impaired users. I decided to use phone vibrations, but unfortunately most phones have very cheap vibration motors that can't react fast and sharp enough. That's why this project was in a drawer for years, while I was consistently failing, trying to come up with a DIY vibrating/poking wearables and electrifying myself with 1000 V spikes.

Luckily, Neosensory generously sent me their Buzz device to participate in their developer's contest. This publication is the result of my acquaintance with the Buzz and my appreciation to Neosensory and it's co-founder David Eagleman, who's books and research defined a big part of this life-long communication research and made this app possible.

Contest video

Android App

During the last two weeks I had to learn Java and Android development from scratch in order to accomplish this project in time. Luckily, I have experience coding in other languages, but it still took a week to figure out everything.

Then I had to develop a tiny text-adventure "game engine" that allows me to create new levels by just adding json files in the levels folder. There were some troubles and bottlenecks in the Bluetooth communication, but I managed to play with batching and precise timing to make the Buzz work (see the Buzz class in the project code).

The app is still not reviewed in the Google Play store, so for now you will need to install the apk file located in the app/release folder of the repo to try the game. (update: the game was approved in Google Play market).

The new Haptapp Uploader app is an accessible text-only-based quest that takes advantage of user's imagination and spatial memory through a technique called "Memory Palace". It builds a space in user's brain where each letter has it's own room and each room has a unique easy-to-understand vibration pattern assigned to it. This gives a basic motion and space-based intuition to a user who wants to learn reading from haptic feedback.

There is still a lot to polish in the code, but the contest and the deadline made me push through and release a playable product. I still have to write more levels though.

How the uploading works

Memory palace is a well-known and very effective memorizing technique used by most memory champions. If you have ever seen people who can memorize a deck of cards, they are probably using a memory palace. Haptapp defines a memory palace in user's brain along with a set of movements and timing restrictions inside of it in a way that one person can understand where another person went in the palace by just listening to the timing of door claps. The game forces the user to repeatedly guess location of an energy ball in the palace, which imprints the map in spatial memory much more quickly than it's possible with muscle memory.

What's happening at the background is that the game actually trains a new neural network in the user's brain. The vibration patterns are the input data, the room names are the labels, user acts as a model that attempts to predict room names, and the dopamine released from passing a level acts as back propagation tool that gratifies the neurons for correct predictions.

The spatial map is only used at the beginning of the learning process, as a way to self-check predictions and self-correct without a teacher, same as children can add numbers multiple times when they forgot the result of a multiplication. Later, the rhythms will settle in other kinds of memory and will not require the memory palace. Further in the future, if the vibrations are played fast enough, a person can learn perceiving whole words as recognizable rhythms.

What is Haptapp for

While haptapp may not be a good replacement for day-to-day human-to-human communications, it can be useful at some other applications such as background speed reading, reading for blind, feeling speech for deaf.

Haptapp also allows short messaging with devices and wearables. Any device can have a vibration sensor instead of buttons that listens for rhythmical patterns. For example, a car stereo can be operated blindly by tapping the steering wheel in a specific pattern to play, stop songs or change the volume.

Another interesting observation: people who learned haptapp discovered that they can recognize familiar rhythmical patterns in dog barks and cricket chirping. Apparently, species that don't have much vocal variation capabilities are forced to use rhythmical communication. This discovery started a parallel research of a possible human-to-dog communication using a special haptapp-like word map designed specifically for communication with other species.

Once Neuralink Is available to public, haptapp can be used for fast sending and receiving text through direct neural stimulation.

In the future, we may discover that children, who were exposed to haptapp early may think in a different way. As thinking is mostly a process driven by language, learning a non-verbal communication method may lead to non-verbal spatial thinking, which could be much faster and deeper than ordinary thinking. I believe that haptapp could evolve into some kind of an Assembly language for brains that will develop an effective link between spatial memory and input/output logic before children acquire a language. This can potentially make it possible to "grow" or develop computer-like abilities inside a human brain and excel as species.

Please subscribe to haptapp newsletter if you are interested in this technology.