Gesture Tetris

Gesture Tetris is novel implementation of classic Tetris game. Instead of keyboard it is controlled directly by user hand swiping in the air in front of the infrared gesture sensor. Project feature MAX32620 microcontroller and MAX25405 Gesture Sensor which together creates perfect base for gesture detection projects and also offers very easy device build and assembly. This bundle is available as commercially available evaluation kit from Analog Devices and this project is rather smart utilization of this kit instead of complex breadboard or other DIY circuit. It is very easy to build the kit and run the game. The software side of the game is implemented as an online game which uses modern JavaScript API for communicating with the physical device and do not require any software installed on computer. Naturally work on Windows, Linux and Mac.

The game is ideal for people who can’t use fingers to control mouse and keyboard or can’t do that precisely enough. Game allows playing with much less precise hand movements. While it seems pretty easy, actually gameplay is slightly challenging because gesture detection takes some time and there are latencies.

See it in the action!

FLICKER WARNING: Sensor used in project emits Infra-Red light which my camera was unable to filter at some moments. For this reason, there is flickering in the video.

Building Device

Device is very easy to build. Simplified instructions to build are also directly present https://gesture-tetris.misaz.cz/website where the game is hosted.

• Connect MAX25405 Feather Shield Board with MAX32620FTHR board which both comes in MAX25405 evaluation kit. Remember to check that you connect 16-pinheader side to 16-pinsocket and 12-pinheader to 12-pinsocket. Rotating the board and connecting it wrong way can damage the device.

• Connect Long narrow MAX25405 Sensor Board with shield assembled in previous step. For connection use ribbon cable which you received as part MAX25405 evaluation kit. Connector has lock, so there is no way to connect it wrong way. Connect sensor to SENSOR 1 slot.

• Connect 3.3V DC power adapter to wall outlet and jack to the evaluation kit. Evaluation kit comes with US plug. If you are outside US, you need to buy 3.3V at least 1A DC power supply with compatible jack.

• Connect USB cable between PC and assembled device.

• Mount long narrow sensor board to static position and sensor must be oriented to sense area where you will move your hands to make gestures.

That’s all. Now you can open https://gesture-tetris.misaz.cz/website, click connect, select kit and swipe hand up for starting the game.

Game Control

Game is controlled using following gestures:

• Move hand from bottom to top: starts the game
• Move hand from left to right: moves currently falling block one position to right
• Move hand from right to left: moves currently falling block one position to left
• Move hand from top to bottom: speeds up the process of block falling and let it fall immediately
• Rotate hand clockwise: rotates currently falling block clockwise
• Rotate hand anti-clockwise: rotates currently falling block anti-clockwise

Block diagram

The diagram of hardware connections is the following:

The parts in the dashed rectangle come as part of MAX25405EVKIT including microcontroller board. The only external part you need is – computer.

Firmware

Device relies on MAX25405EVKIT firmware which contains driver for the MAX25405 sensor and contains advanced gesture detection algorithm.

Software

The most advanced part of development was software side which runs in web browser. It interfaces with device using serial port. This port is virtual UART over USB port and on the device side it is managed by firmware. Gesture Detection Firmware is preprogramed in the microcontroller shipped with MAX25405EVKIT. Firmware was originally designed to be used with evaluation kit software GUI which look as follows.

It is nice software which can configure sensor and gather data from it. But in this project, I used it differently and I attached my program to the same API which use this software. Instead of showing RAW data I instead interpreted detected gestures in browser and bind Tetris actions to the gestures.

The game itself is implemented in TypeScript which transpiles to JavaScript.

Sensor Operation

Sensor which is the center of evaluation kit board is sensitive to infra-red light. It measures amount of infrared light received. It does not work alone. It requires some infra-red (IR) light source. This is done using 4 IR LEDs which are near sensor. Sensor controls them and it periodically switch them on and off regularly and generates pattern which it use for distinguishing from external IR light sources. When there is no object in the field of view of the device, light is emitted but do not reflect and “fly” away. Sensor sense only low amount of light returned. In opposition, when there is object in the field of view, light will “fly” to the object, then reflect and amount of light which come back to the sensor is higher.

MAX25405 Sensor do not contain only single sensor, but instead it contains matrix of 10 x 6 such sensors. It contains lens which sets the angle of view. The data which comes from sensor are not single value, but rather it is small image. We can collect several tens of this images per second and analyse behaviour of the object in front of sensor. When the hand will move, the direction of light reflection will change, and this will be visible on small images collected from sensor. This is exactly how firmware on MAX32620 MCU works. Data from sensors are preprocessed using filter and at the end they are passed to several detection engines which detects target gestures. Some detections are trivial, some utilizes machine learning techniques.

Benefit of this approach in comparison with Time-of-Flight approach is that just sensing IR light instead of time of arrival is easier and more time-deterministic. Additionally, it can be done faster which results to much higher frames-per-second rate. Depending on configuration it can be between 60 – 120 FPS in most standard configurations. In project it is configured to approx. 75 FPS.

Hardware improvements

For basic usage the sensor board can be attached for example to top edge of laptop by sticky tape. But better solution is to make stand for it. I made my own DIY stand based on Meccano-style building kit. It is not exactly Meccano but instead it is Merkur building set, which is available in my country. Difference between Merkur and Meccano is mainly in pitch between mounting holes. Merkur use metric 10mm pitch and M3.5 screws.

Closing

In this article I described my gesture controlled online game. You can play it on https://gesture-tetris.misaz.cz/and the only hardware thing you need, is MAX25405EVKIT. The benefit of this project is the simplicity of hardware build. You need just one kit and no breadboard builds, no PCB, no soldering. It is just smart utilization of this kit. It should be very easy to build and play for (not only) disabled people. In the Resources section below you can find source code of Tetris web game, but unless you want to edit sources you do not need them because game is deployed at https://gesture-tetris.misaz.cz/.

This is all for this project. Thank you for reading it as far and have a nice day.