Bartosz Marcinkowski's Open Source Gest Tool Turns Hand Gestures Into Mouse Movements, Interactions

Developer Bartosz Marcinkowski has released an open source tool dubbed gest, which turns hand gestures captured by a webcam into mouse controls — positioned as an ideal partner for speech recognition systems like Talon to boost accessibility.

"For health related reasons, I had to stop using a mouse and a keyboard," Marcinkowski explains. "Talon allowed me to type with my voice and move the cursor with my eyes. This project was started to complement this setup with hand gestures."

"The project is in an early stage of development. I use it on daily basis, so it should be good enough for some. What is implemented: Pinching gesture recognition, in one hand orientation; heatmap output, separate for left and right hand, indicating pinched point position; demo for testing recognition models; example script for simulating mouse clicks and scrolling; scripts for producing and reviewing training data."

The Python-based gesture recognition system ties into a webcam and allows for mouse point control, scrolling, left- and right-clicking - effectively covering the majority of mouse controls required for general-purpose use.

It does, however, come with a caveat about the data on which it is trained. "The gesture recognition model was trained on images of my hands, taken with my hardware in my working environment, so it is probably heavily biased," Marcinkowski warns. "I hope people who want to use it, but recognition quality prevents them from it, would capture some images of their hands using included tooling and donate it to the project, so that over time it works well for everyone."

Gest is now available from GitHub under the GNU General Public License 3.0.