Building a Touchscreen Mouse with Shortcuts

This project was inspired by a kickstarter scam that in hindsight a bit too good to be true. The product that was being advertised was a mouse that was covered with a flexible touchscreen OLED that could be programed with shortcuts to any number of applications. It seemed like a novel and interesting idea (even if not fully practical) where they seemingly showed functioning prototypes as shown below.

When the project did eventually fall through; I decided why not try to make the best of the situation and build my own. The goal of this project was to make a functioning mouse that implemented a touchscreen display that could be programed with custom shortcuts for various applications.

This is also a story, in a continually growing large part, about scope creep. When I started this project, the plan was to Frankenstein together an existing mouse with a touchscreen for shortcuts. Maybe a simple USB data splitter to handle both the mouse signal and shortcuts from the controller…

Learning How a Mouse Works

The Basics

First and foremost, this needs to be a functional computer mouse so we need to figure out how they work. Essentially all mice that we use today have optical sensors that are used to detect the movement of the mouse. They work by taking consecutive images of the surface under the sensor and comparing each image to the previous image then calculating the relative motion of the mouse based on the offset. That x and y offset is then sent to the computer which then turns that data into the motion of your cursor.

Some of these chips also take input from the buttons and scroll wheel and send those inputs as well. So, theoretically, it should be pretty simple to create a functional mouse from a torn down mouse or two… right? Well it turns out that something as simple, common, and cheap as a computer mouse can make you feel like an idiot.

A Teardown or Three

The most fun part of reverse engineering something is always the teardown; so I bought and tore apart a few inexpensive options. Two were standard wired USB mice and the third was bluetooth.

Each of the mice had a different optical sensor; all doing essentially the same thing but in different form factors and resolutions. Unfortunately, not all of the data sheets were available for the sensors which meant trying to repurpose these into my own circuit wouldn’t have worked. Even worse is that one of the sensors (mouse on the right) used PS/2 communication protocol with is mostly obsolete and would have really limited its functionality.

The first one (on the left) initially seemed like it was going to helpful in testing because I was able to find its data sheet online. This chip takes its data and uses the Human Interface Device (HID) standard to talk directly to the computer via USB without the need for an additional microcontroller.

While that sounds helpful, it’s actually an issue once you try to integrate it to work with a microcontroller like I am. The reason is that the controller would require extra software packages that can first translate this to data that is usable by the microcontroller. I’ll go more in depth on this in the next section.

The last was a bluetooth mouse. Overall this wasn’t much different from the first except some battery charging circuitry and another IC for processing and sending the data over BT. Unfortunately the sensor on this one was proprietary with no data sheet to be found. So without being able to see inside the chip, this ended up being useless.

Talking to the Mouse

All I wanted was data, but all I got was pain. I thought this was going to be the easy part of the project but quickly found myself going off the deep end trying to get this to work.

As I quickly learned, there are two primary ways that the mouse and computer communicate; PS/2 and HID. PS/2 is an old standard that isn’t used anymore while the HID standard is used by essentially all mice and other devices like keyboards. I won’t get into these much more than this for now because it’s easy to get lost with how the two protocols actually work. Plus, I don’t understand them well enough to explain them either.

To start actually getting some data from the sensor it needs to be 1) connected to a power source and 2) connected to something that cares to listen. In this case I ran the testing on an ESP32 devkit board. I’ll be talking about it more later, but this will ultimately be the brains of this project.

Like I mentioned above, the microcontroller has to be able to convert from HID to more usable data. While this would not ultimately be practical for the final version, I still wanted to test getting and sending data. Fortunately, I found a couple of helpful libraries that did just that so that I could test getting sensor data. If you’re interested in learning more about how this communication works then I’d recommend digging through these repositories.

Github Repository: ESP32 USB Soft Host

I was able to use this setup to do some initial testing on the code and confirm that it would be feasible to integrate the controller with an optical sensor and move the cursor.

Going Straight to the Source

Note to the reader: I’m a mechanical engineer so we are now deep in uncharted territory. When I mentioned that this was in part a story of scope creep, this is what I meant. Originally the plan was to solely leverage breakout boards and jam them all into the mouse.

After some searching, I found that someone had actually created a repository to communicate with an old mouse sensor. Specifically, the ADNS-5050 optical sensor. I figured this was my best option to avoid having to try to write my own program to gather data from the sensor.

Schematic

In order to keep things relatively simple for my own sake, I created a basic breakout board for the sensor. The schematic below matches what is outlined in the ADNS-5050 Data Sheet.

PCB Manufacturing

I want to give a big shoutout to PCBWay for helping with this project and providing the breakout boards for the sensor testing. If you’re looking to test your own PCB design I’d highly recommend giving them a try.

This was my first time designing a custom PCB so I was a bit intimidated going into this but they made it super easy even for a beginner. Since I had some components already on had so I decided to use a basic thru hole design and do the soldering myself. The boards all came in exactly to spec and the best part was that they arrived within 2 weeks of placing my order so I didn't have to wait long!

The Brains

As mentioned above, I’m using an ESP32 development board for this project. The built in bluetooth is perfect for this project so there’s no need for another module.

Using the breakout board, we are finally able to get movement data directly from the sensor to the microcontroller. Using another repository, that data can be used to control the computer over bluetooth.

Github Repository: ESP32 BLE Mouse

At this point, it’s technically a functional bluetooth mouse!

Touchscreen

The defining feature of this project is the touchscreen which needs to be reasonably sized to fit inside the mouse. There are a number of options out there for small touchscreens so it’s just a matter of finding the best fit. I ordered and tested a couple different options before settling on one for prototyping; a 2.8″ TFT LCD. This screen has a nice small form factor and, importantly, capacitive touch capability (this product has been updated to a new version since I bought mine).

I decided to go for capacitive touch instead of resistive touch because I wanted to limit the amount of pressure that would be needed to trigger the shortcuts. With the mouse intentionally having a low friction base to allow for it to slide easily, pressing too hard on an angle may cause unintended motion.

Shortcuts

To begin with I wanted to create a simple user interface that allowed for the selection of different applications which would then reveal the specific shortcuts. I’m not much of a programmer so simple is the keyword here. The apps and shortcuts are each given an icon that is displayed as a part of a grid on the screen. When an app is selected, it will display a grid of shortcuts. When a shortcut is selected it will send the command to the computer over the USB interface.

As a basic proof of concept here I created simple icons to click that send commands or text to the screen.

Scrolling

So far so good but there’s still something missing. In order for this to really work as a mouse, it has to be able to scroll up and down. The obvious problem here is that the screen is going to be where there would normally be a scroll wheel. That means that the program will also have to simulate this in some way.

To keep things simple here I’m just going to use up and down arrows for scrolling. Trying to incorporate a trackpad style of scrolling would be much more complicated than needed for this project.

Assembly

In order to have a standalone mouse, I made a basic printed base to hold all of the electronics. The mouse also needs a battery on board since it’s wireless. So in addition to the battery; a charging board and 3.3V to 5V converter (the mouse sensor requires 5V) were added.

I used a bit of electrical tape over the sensor to prevent ambient light from impacting the readings.

Since this is really a proof of concept, I decided not to design an outer shell to hide the components.

The Finished Product

I’ve wanted to do this project for a couple years so I’m excited to have finally have completed it.

So it works, but is it useful? Yes and no.

Going into this project I knew that a touchscreen mouse wouldn’t necessarily be a practical device but that doesn’t mean it’s not still interesting. It’s a fun concept and you can add shortcuts for any application that you’d like which can be helpful. At the same time what makes a mouse so useful as a peripheral is that you don’t have do actively look at it to use it. In practice, if you already have muscle memory on where keyboard shortcuts are, then the mouse may be slower while re-learning shortcut locations.

The form factor of this mouse could realistically be much smaller using a PCB that integrated all the separate boards used into one. I also decided that trying to make this version of the mouse look more polished wasn’t worth the effort.

Even given all the additional parts, the overall size of the prototype is comparable to an average mouse. With some refinement, I think it would be totally feasible to make a more ergonomic version that would still maintain a small form factor.

I’m making the code for this project open source for anyone who’s interested in trying to make their own or repurpose portions of it for other projects. Due to some instability issues, I decided to comment out the keyboard portion of the code but left it in for anyone who wants to test it (this may be in part because of my OS).

Github Repository: Touchscreen Mouse

What’s Next?

Interested in helping expand on this project? Let’s team up! I’m happy with how version 1 came out, but it is far from being refined and usable as a daily driver. If you’re interested in a version 2 and feel that you’d be able to help improve the next version, then please reach out. That could be anything from the code, PCB design, or CAD.

I think there’s potential to do some pretty cool things with this concept; like adding multimedia controls or an on board game controlled by the sensor data. I’d also like to find an alternative optical sensor that runs on 3.3V and isn’t obsolete.

Alternatively if you just have some ideas for how to improve the mouse; send me a message or put a comment below. Thanks!

One last thing: I also have a website and I'd really appreciate if you took a minute to check it out! Sam's Workbench.