This Game Makes You Follow a Stickman's Poses to Win

A previous project

Back in February of 2022, Lorraine Underwood from element14 Presents had created a new iteration of a previous window display project by using pose detection to detect whether a person standing close to the window was raising their left, right, or both arms. It worked great, as the TensorFlow Lite model powering the entire system was able to quickly and accurately determine poses.

The idea

Based on this previous experience with embedded machine learning and computer vision, Underwood had the idea to go even further with the concept by not only reacting to what the user is doing, but also give instructions and check their actions in return. This game would be similar to dancing games that rely on people stepping on the correct directional pad whenever the screen presents it, except that this one would integrate the aforementioned pose detection TensorFlow Lite model to capture this data.

Recognizing poses with TensorFlow

The project runs on a single Raspberry Pi 4 with the 64-bit version of Raspberry Pi OS installed since TensorFlow requires it. After downloading the pose estimation example from GitHub and setting up the folder, Underwood quickly inspected the demo code and found that it detects all people present in the video frame along with the count/confidence level of the presence of various body parts for each.

Initial trials

The idea behind the game itself is to show the player one of several images that showcases the pose they should make. This could include raising a hand, lifting a leg, or a combination. The code then displays the correct image, waits a little bit, and then continually checks if the player has successfully performed the action.

Fixing the bugs

One of the main issues Underwood encountered while modifying and testing the existing example code was that showing images/video slowed down the Raspberry Pi 4 far too much. So, as a quick solution, she removed the call to start a new feh process and instead replaced it with another imshow() function call. Another problem she found involved clearing the framebuffer between pose updates so that duplicate points couldn't be won from an identical pose, such as raising the left hand and then being instructed to immediately raise your left hand again.

Playing the game

With most of the problems now fixed, Underwood tried out her new game by first trying to rack up the highest score possible in her living room and then having her family test it out. Although certain background objects did have a tendency to be misclassified as a leg or arm, the overall game worked quite well.

Future possibilities

In its current form, the hardware simply consists of a Raspberry Pi 4, webcam, and some kind of television, but the hope is to swap out this display for something more embedded, including a grid of individually addressable RGB LEDs. Another idea was to get the stickman to follow the user's movements on this LED matrix and essentially make it dance just like the person in front of it. To see more about this pose estimation game, you can watch Underwood's video here on the element14 Presents YouTube channel.