team AURA:

•

•

•

Published April 8, 2025

Project AURA-Architectural User-responsive Reactive Assembly

A gesture-controlled model of UoM’s Engineering Building using Pi 5 + ML — wave your hand, and the architecture moves.

IntermediateWork in progress24 hours534

Project AURA-Architectural User-responsive Reactive Assembly

Things used in this project

Hardware components

Raspberry Pi 5

The heart of the whole operation

Raspberry Pi Pico

We used a pi pico to control the motors as it was a method our group were familiar with, had we not been time limited we could have just used the Pi 5.

DC motor (generic)

This Motor was used to rotate the entire model.

Servo Module (Generic)

2 Servos were used to open the model up, showing an interior view. We used servo's here due to requiring more precise control.

Rechargeable Battery, 4.8 V

any power supply is suitable provided it has an accompanying voltage regulation circuit to ~5v

1N4007 – High Voltage, High Current Rated Diode

protection diode for power supply to pico

Logic Level FET P-Channel

p channel mosfet for switching circuit for dc motor

Resistor 1k ohm

any resistor above 220 ohm is suitable for pull down resistors for which three are required for this project

Software apps and online services

Raspberry Pi Raspbian

gpiozero

Story

We’ve developed an interactive, gesture-controlled scale model of the University of Manchester’s Engineering Building, powered by machine learning and embedded systems.

Our model responds to hand gestures in real-time:

Swipe left → left side of the building opens

Swipe right → right side opens

Raise both arms in a T-pose → the entire building rotates

This touchless control system allows users to interact with a physical structure in a completely new way, offering endless possibilities for educational tools, exhibitions, and smart environments.

💡 Why We Built It

We set out to explore how gesture recognition and embedded machine learning can be used to control physical models. Our goal was to demonstrate how these technologies can be leveraged to create intuitive, immersive experiences where users can engage with models or environments in a more natural, hands-free way.

In many scenarios, traditional interaction methods—like buttons or screens—can be limiting. By incorporating gesture control, we enable users to engage with a system without touching it, adding flexibility and accessibility. This concept has applications in:

Smart architecture: Buildings or spaces that react to human input.

Interactive exhibits: Where users can control models or display systems via simple gestures.

Public spaces: For contactless interaction with technology, ensuring ease of use and accessibility.

⚙️ How It Works

Gesture RecognitionThe system uses a Raspberry Pi 5 with a camera to detect gestures in real time. A TensorFlow Lite model is employed for efficient gesture classification. We trained the model to recognize specific hand gestures:

Swipe left: Initiates opening of the left side of the model.

Swipe right: Triggers the right side to open.

T-pose: Commands the model to rotate around its central axis.

Actuation (Raspberry Pi Pico)The Raspberry Pi Pico acts as the real-time controller for the servo motors that physically manipulate the model:

Two servo motors control the opening of the left and right sides of the building.

One central servo motor controls the rotation of the model when the T-pose gesture is detected.

This division of labor between the Pi 5 (for gesture detection) and the Pico (for real-time motor control) ensures the system operates efficiently with minimal latency.

🛠️ Tec h Stack

Raspberry Pi 5:

Runs the gesture detection model using TensorFlow Lite for efficient, edge-based machine learning inference

Uses OpenCV to process camera input and detect the gestures.

TensorFlow Lite: A lightweight version of TensorFlow, optimized for edge devices like the Raspberry Pi. It enables efficient real-time gesture recognition.

Finite State Machine: Tracks user gestures and interprets the sequence of movements to determine the appropriate model response. The FSM ensures actions only occur when the correct gesture is performed.

Raspberry Pi Pico: A microcontroller that interfaces with the servo motors for real-time control of the model’s movement.

Servo Motors: The physical actuators that control the opening of the model and its rotation. These are powered and controlled via the Raspberry Pi Pico.

Custom-Designed Model: The UoM Engineering Building model was handcrafted using foamboard and 3D-printed parts to create a functional representation of the structure. The model was designed with three key moving components:

Left side opening

Right side opening

Central rotation mechanism

🧠 Wha t We Learned

Real-time machine learning on embedded devices: Using the Raspberry Pi 5 for gesture recognition in real-time, paired with TensorFlow Lite, proved to be both efficient and responsive.

Servo control: Managing real-time actuation with the Raspberry Pi Pico and coordinating the servo movements was challenging but demonstrated the power of using low-cost hardware to create interactive models.

Gesture sensitivity and calibration: Fine-tuning the gesture detection required iterative testing to balance accuracy and responsiveness in diverse lighting conditions and environments.

🎯 Pot ential Applications

Project AURA shows the potential of gesture-based control systems for applications in various fields:

Interactive Exhibitions: Museums, science centers, and educational settings could use gesture-controlled models to engage visitors and demonstrate complex systems in a hands-on way.

Smart Buildings: This concept could evolve into a system where users control architectural elements — like blinds, windows, or even rooms — simply by gesture.

Public Spaces: In environments like malls, public transport, or smart cities, users could interact with digital displays, models, or even physical structures through gestures, reducing the need for physical touchpoints.

🌟 The Takeaway

Project AURA demonstrates that machine learning and embedded systems can be integrated seamlessly to create a hands-free, interactive experience. By using simple gestures to control a physical model, we provide a glimpse into the future of intuitive, responsive architecture.

We believe this project is just the beginning, and with further development, it could expand into applications in smart cities, accessible technology, and educational tools.

Schematics

Code

from modlib.apps import Annotator
from modlib.devices import AiCamera
from modlib.models.zoo import Posenet
import numpy as np
from math import sqrt
from gpiozero import Servo
from gpiozero import LED
import time

#GPIO pin setup for digital communication with the raspberry pi pico
servoRight = LED(2)
servoLeft = LED(3) 
motorswitch = LED(4) 

#setting up and deploying pre-trained ai model
device = AiCamera()
model = Posenet()
device.deploy(model)

#setup annotation utility to dray pose landmarks
annotator = Annotator()

#start reading video frames
with device as stream:
    for frame in stream:
        detections = frame.detections[frame.detections.confidence > 0.5]
        poses = frame.detections  # always called frame.detections, depends on model not object
        
        poses.keypoints[0] # this will store 34 values of the first person detected
        person0 = poses.keypoints[0]
        
        #extract normaalised co-ordinates of hips and wrists for gesture detection
        leftHipx = person0[23]
        leftHipy = person0[22]

        rightWristy = person0[18]
        rightWristx = person0[19]

        leftWristy = person0[16]
        leftWristx = person0[17]

        #initialise gesture flags
        t_pose = False
        leftturn = False
        rightturn = False
        
        #calculate horizontal distance between wrists 
        leftDiffsx = abs(rightWristx - leftWristx)

        #if wrists are far enough apart assume t-pose
        if leftDiffsx > 0.45:
            print("T pose detected, model opening...")
            t_pose = True
        
        #calculate horizontal distance between left wrist and hip
        leftArmDist = abs(leftWristx - leftHipx)
    
        #if right arm stretched out assume right turn
        if leftArmDist > 0.16:
            if t_pose == False:
                print("model rotating right...")
                leftturn = True
                
        #if right arm stretched out and not currently rotating right or t pose, rotate left
        rightArmDist = abs(rightWristx - leftHipx)
        if rightArmDist > 0.16:
            if t_pose == False:
                if leftturn == False:
                    print("model rotating left...")
                    rightturn = True

        #based on gestures, trigger respective outputs
        if t_pose :
            motorswitch.on()
        else :
            motorswitch.off()

        if leftturn :
            servoLeft.on()
        else :
            servoLeft.off()

        if rightturn :
            servoRight.on()
        else :
            servoRight.off()
            
            
        # Annotate and display the frame
        annotator.annotate_poses(frame, detections)
        frame.display()

#include <Servo.h>

Servo rightServo;
Servo leftServo;
const int ledPin = 25;    // Onboard LED for Pico
const int rightPin = 2;   // GP2 (physical pin 4)
const int leftPin = 3;    // GP3 (physical pin 5)
const int inp1 = 0;       // Right servo control (active HIGH)
const int inp2 = 1;       // Left servo control (active HIGH)

// Servo states (FSM)
bool rightServoState = false;  // false=0°, true=180°
bool leftServoState = false;   // false=0°, true=180°

// Previous input states for edge detection
bool prevInp1 = LOW;
bool prevInp2 = LOW;

void setup() {
  pinMode(ledPin, OUTPUT);
  rightServo.attach(rightPin);
  leftServo.attach(leftPin);
  pinMode(inp1, INPUT);  // pull low resistor, input from Pi5 
  pinMode(inp2, INPUT);  // pull low resistor, input from Pi5 
  
  // Initialize
  rightServo.write(0);
  leftServo.write(0);
  digitalWrite(ledPin, LOW);
}

void loop() {
  // Read current inputs
  bool currentInp1 = digitalRead(inp1);
  bool currentInp2 = digitalRead(inp2);

  // Right servo control (trigger on rising edge)
  if (currentInp1 == HIGH && prevInp1 == LOW) {
    rightServoState = !rightServoState; // Toggle state
    
    if (rightServoState) {  // If true, set to 90°
      rightServo.write(90);
      digitalWrite(ledPin, HIGH);
    } else {  // If false, set to 0°
      rightServo.write(0);
      digitalWrite(ledPin, LOW);
    }
  }
  prevInp1 = currentInp1;

  // Left servo control (trigger on rising edge)
  if (currentInp2 == HIGH && prevInp2 == LOW) {
    leftServoState = !leftServoState; // Toggle state
    
    if (leftServoState) {  // If true, set to 90°
      leftServo.write(90);
    } else {  // If false, set to 0°
      leftServo.write(0);
    }
  }
  prevInp2 = currentInp2;

  delay(10); // Minimal delay to prevent busy-waiting
}

Credits

Comments

Please log in or sign up to comment.

Project AURA-Architectural User-responsive Reactive Assembly