Raspberry Pi Sign Language Interpreter

For the University of Manchester’s Hack-a-Bot robotics society event, we set out to build something meaningful—something that could genuinely help people. With over 100 students participating and only 24 hours to complete our projects, we wanted our project to stand out not just for its tech, but for its purpose.

Our idea? A system that could recognize sign language letters using just a Raspberry Pi and the Sony IMX500 AI camera. No external servers. No cloud processing. Just pure edge AI.

We initially experimented with the MediaPipe library for hand tracking. While it gave us a good starting point, it wasn’t accurate or flexible enough for the specific task we had in mind. So, we pivoted—and that’s when the real learning began.

We built a custom Machine Learning model using TensorFlow. To train it, first, we experimented with already existing data sets. After realising that these could not really serve our purpose, we collected our own dataset, capturing images of our hands forming different sign language letters. It was a repetitive process—photograph, label, train, tweak—but it allowed us to control the quality and specificity of the data.

The technical pipeline required several key steps:

• Designing and training a custom classifier
• Applying quantization to make the model lightweight
• Compiling and packaging it correctly to run directly on the IMX500 camera

The best part about this project, and this camera with AI integration was that the camera itself did the heavy lifting. The model ran locally on the Sony IMX500, eliminating the need for a separate GPU or cloud processing—everything happened right there, in real-time, on the device.

If we were to take the project further, we’d look into implementing hand segmentation to improve prediction accuracy. This would help the model better isolate the hand from the background, reducing noise and improving detection under varied conditions.

In the end, this project was more than a hackathon entry. It was a small step toward making technology more accessible and useful—one sign at a time.