Minecraft Gesture Controller

Move your body to play Minecraft! What!! Yes. Check the video above for a demo :)

This tutorial will show you how to make your very own gesture game controller for Minecraft (or your other fav. computer game). Move your hand(s) to walk/run/jump, look around, and attack* all the things!

Let's get started! Grab yourself a Circuit Playground Express, snag my program code, and get shakin' to play Minecraft in (srsly) the most fun way ever! :D

Read time: 20 min

Build Time: ~ 2 hours

Cost: ~$30

*It is a biiiiit tricky to attack moving things (like monsters), so be careful in survival mode! Or use this to challenge your skills :)

Materials & Tools!

Build the Glove Controller! (Pt. 1)

You can make the gesture controller without the glove, but the glove controller makes it easier to play, keeps the CPX in the same orientation (very important), and means you can use your fingers as added controls!

1. Cut rectangles of conductive fabric for the finger pads (~ 0.5 in. x 1 in.).

2. Use regular thread to sew the conductive fabric pads onto each of the glove fingers.

Suggested to use a highlighter or other pen to avoid sewing the two sides of the glove together (learn from my mistakes bbies).

3. Attach CPX to the glove with velcro squares.

4. Use an alligator clip or insulated wire to connect the CPX ground ("GND") to the thumb pad.

5. Stitch conductive thread from the CPX capacitive touch pads (A1, A2, A3 & A4) to each of the four fingers.

6. If you have a multimeter, check continuity between the CPX pins and the conductive thread pads.

Plan Out the Game Controller!

First! What do we need to do to control Minecraft (or another awesome game)?

This is a super helpful & fun lesson in Design Thinking, but you can skip this if you want to just use my controls. You can always come back here later if you want to make changes later :D

1. Determine (crucial) game controls.

Note: Start simple! Figure out the most important controls for the game and start there. You can always add more later.

Here are the controls that I wanted to use while playing Minecraft.. in creative mode :) (you can use the same ones or customize your own controller!):

Movement:

• Walk forward: W key
• Run: Ctrl + W
• Jump: Space bar
• Look Left & Right: Mouse rotate
• Walk backward: S key

Actions:

• Attack: Mouse Left Click
• Place Block/Push/Open: Mouse Right Click
• Inventory: E key
• Escape: ESC key

2. Decide how you want to use gestures and/or the finger pads to trigger these controls. Recommended to sketch out your plan.

Here is my design thought process:

I've always wanted to feel like I was actually *in* a game, so I went the "cheap VR" route and used gestures to control basic movements. For walking, I went the "let's move my arms like I'm walking" route, which easily transitioned into running and jumping by increasing the speed of motion.

To make it easy to place a block or exchange items, I decided to use an "awkward handshake" motion.

Turning was a bit of a challenge, but my goal was to be able to look around by moving my hands in the direction I wanted to look.

Attack became the pointer finger pad, inventory the middle finger pad (which I ended up removing), Escape the ring finger pad, and the pinky finger pad to let me to walk backwards.

Again, you can keep these same controls or design your own :D

Let's Get Programming: Set Up the CPX!

1. If you're using Windows, download the Adafruit Windows Drivers here.

2. Download & save the latest CPX Circuit Python UF2 file.

3. Plug in the CPX with a USB cable (make sure it has data transfer capabilities).

4. Double-click the reset button on the CPX.

The LEDs should turn green. If they are red, it means something is wrong with data transfer to the CPX -- check the USB cable, try another USB port on your computer, or try the trusty "unplug and plug back in" method.

5. On your computer, you will see a new disk drive called "CPLAYBOOT".

6. Drag the CPX Circuit Python UF2 file onto the disk drive.

7. The "CPLAYBOOT" drive will disappear and be replaced with "CIRCUITPY".

Add All the Libraries!

Libraries let us access all sorts of special functions for the CPX without having to do a ton of programming.. hooray open-source! This install will download most of the standard MicroPython libraries*. Peruse them at your leisure to learn about more cool things you can do!

1. Download and save the Adafruit Circuit Python Library Bundle Release from here.

2. Unzip the folder, open the first folder, and copy the lib folder onto the "CIRCUITPY" drive.

*It is unlikely that you'll run out of space since the CPX comes with at least 2MB of Flash storage. But, if you do end up needing more space, you can revisit the libraries and remove the ones you don't need. If you mess 'em up, just copy and paste the lib folder again.

Writing the Controller Code!

The CPX has an on-board compiler, which means you can program it in (pretty much) any language you want! I opted for MicroPython, a version of Python for microcontrollers, 'cause Python is awesome.

Read this step if you want to understand how the program works (definitely suggested) or if you want to modify it to design your own version.

You can find the source code in the "Code" Attachments at the end of this tutorial.

1. To do the things we want with our controller, we need the following MicroPython libraries:

LIS3DH accelerometer

• This allows us to use motion to trigger various things.

Human Interface Device ("HID") keyboard

• This library allows us to control the keyboard!

HID mouse

• This library means we can control the mouse!

CPX capacitive touch

• This library lets us use the capacitive touch feature on the CPX, hooray!
• A couple of other libraries to make our lives easier: time, busio, and board.

2. Configure and initialize the libraries.

Assign variables for the keyboard, mouse, and accelerometer objects. Select a range for the accelerometer.

3. Write short functions for each of the controls.

The motion controls can be tricky. Do some initial testing with the accelerometer by printing the values in a serial monitor (in the source code, go to the __main__ function and uncomment the two debugging lines). This will help you to determine thresholds for walking, running and jumping, looking left and right, and placing objects.

The touch pad triggers are much easier as you are only looking for a capacitive trigger (True/False).

Remember to release all of the keyboard and mouse keys at the end of each function!

Debugging: Seeing What's Up With the CPX Code

If you're familiar with Arduino, you're probably familiar with the Serial Monitor. The CPX has the same feature with a slightly different access point depending on what program you are using.

If you are using Mu it's super easy: the serial console is built in and will automatically detect your board, yay!.

If you are using Idle or another program, follow these steps:

1. Download PuTTY* here.

2. Go to Windows Device Manager and check the serial port number for the CPX (e.g. COM18) -- see Photo 1.

If there are multiple serial ports listed, unplug the CPX and plug it back in to see which one disappears then reappears.

3. Open PuTTY and select "Serial".

4. Enter the serial port number (e.g. COM18) under "Serial line" and the baud rate of 115200 under "Speed".

5. Click Connect!

*PuTTY is a free and open-source SSH and telnet connection program.

Test & Improve

Load the program onto the CPX by dragging and dropping the python file onto the CIRCUITPY drive, then rename the file as "Code.py"

Like pretty much every project, this one will likely be a little wonky when you first get it running. If the touch pads are acting strange, reset the CPX (this recalibrates the capacitive input pins).

Test 1:

• Open up the serial monitor with PuTTY and run the program (CTRL + D)
• Test each of the movement controls (you'll see the mouse moving on the screen and make sure the program doesn't crash as well as the touch pads (which should display relevant text on the serial monitor).

Test 2:

Deploy in Minecraft creative mode! Test the movement and action controls to see if anything breaks or doesn't work as expected (plz keep in mind that this is a prototype)

Update the program based on your testing. Remember, it's OK if it's not perfect, there's always time to make it better!

Have All the Fun!!

You're ready to run through Minecraft!! Just be wary of monsters, it might be a bit tricky to protect yourself..

Supplementing your gesture controller with a keyboard is a good idea if you want play for reals :)

Please like and/or leave a comment if you enjoyed the tutorial! And of course, let me know if you have any comments or questions! Happy Building!

<3, jenfoxbot