Move Over Mouse, Control Your Cursor with This Flute Instead

Why invent a musical mouse?

While human interface devices (HIDs) come in a wide variety of shapes, sizes, and functionalities, we most often end up using the typical keyboard, mouse, and touchscreen, which can get boring. After recognizing this himself, research software engineer Joren Six wanted to move away from the traditional model and come up with a unique method to control a computer. He was inspired by this Reddit post in which a gamer is using a flute to control his aim in a first-person shooter, and instead of crosshairs, Six would move a cursor across the screen.

Setting up the hardware

Because he would need something that could listen for sounds, quickly analyze the recorded audio, and then control a mouse, Six went with the Arduino Nano RP2040 Connect. His code utilizes the board's built-in PDM microphone via the PDM library to continuously receive new data packets and store them in a buffer. Additionally, the USB connection to the host PC allows the RP2040 to act as an HID. Altogether, he called this system the "pitch perfect pointer positioning" solution, or PiPePoPo for short.

Determining frequency

The most challenging aspect of this project was determining the frequency of the captured audio samples, as sound is not just one wave but a combination of many different frequencies at differing amplitudes. In order to find the fundamental frequency, a highly optimized library implementing the YIN algorithm was included. It works by accepting the sampling frequency along with an array of samples before outputting both the pitch and probability of the result. And because the RP2040 lacks a hardware floating-point unit (FPU) owing to its Arm Cortex-M0+ architecture, Six also included an optimized fixed-point math library.

Pointer movements

Once the frequency has been found, the RP2040 Connect's next task is to move the mouse in response to the pitch. Six configured four different, mutually exclusive ranges where the cursor's position will be changed by a set number of pixels to either the left, right, up, or down. A fifth, higher-pitched range causes a click event to be sent before starting a cooldown timer that helps prevent excess clicking.

A software alternative

Six's project works very well on its own, but after realizing that not everyone would have the ability to build the hardware, he also released a Chrome extension that performs a very similar set of actions. It listens to audio through the browser's microphone API and constructs a virtual cursor that moves based on the pitch. When the user wishes to click, the extension simply gets the element at the cursor's coordinates and calls its click() method.

For more information about this project, you can visit Six's write-up here on his blog.