Published September 5, 2024

Mobility Control Assistant

A core device, which can be added to a wheelchair or strapped to the body, allowing control tailored to each user.

AdvancedFull instructions provided417

Honorable Mention Mobility Impairments

Build2gether 2.0 — Inclusive Innovation Challenge

Things used in this project

Hardware components

Nordic Semiconductor nRF52 Development Kit

M5Stack AtomS3

Seeed Studio XIAO ESP32S3 Sense

DFRobot UNIHIKER - IoT Python Programming Single Board Computer with Touchscreen

Software apps and online services

CircuitPython

Hand tools and fabrication machines

3D Printer (generic)

Story

Improved Motion Control

My goal for the BUILD2GETHER 2.0 was to make a device with customisable motion and I succeeded!

This device can be tailored to the user, whether it be someone who suffers from tremors or has limited dexterity.

Some the the Examples are:

The minimum motion can be set to ignore any spasms or tremors, by setting the increments for motion.
The minimum and maximum values can be set to choose how much motion to register or set the maximum movement speed when gaming or controlling a motorised chair.
The deadband option allow you to specify the movement required to start registering. This can also be use in combination with the min/max to use the joystick like a direction pad and only register movement at the extremes.

My Process

The first thing I researched was the best hardware solution to make this device work well and be scalable. After extensive research I decided that the nrf52 devices would have the best battery life and the ESP32 is the most common, so I decided to make my solution non hardware specific. I set out to find the most adaptable coding language and decided on CircuitPython. It is easily customisable, you can find libraries for most hardware and they are easy to add.

I spent about 4 hours, over 2 days figuring out how to add CircuitPython to the Nordic Semiconductor's nRF52840 DK

https://learn.adafruit.com/circuitpython-on-the-nrf52?view=all

Once I had that working, I tested several libraries to decide on the controls and landed on the JoystickXL library because you can add multiple controls and program how they work.

https://github.com/fasteddy516/CircuitPython_JoystickXL

I then proceeded to get the 6-Axis IMU Unit(MPU6886) working as a joystick and how to customise the reactions.

My aim was to allow for the following motion option customisations:

Sampling Rate (Delay) - Allows for setting how often to take movements and remove jitter.
Range of Motion (Distance) - Using a minimum and maximum value allowed the range of motion to be defined.
Dead Band (Zero Center Range) - This is used to set how far from the middle movement is needed to start registering.
Minimum Change (Delta) - This was the most useful setting! It allows the user to set how smooth the motion is, based on the registered increments.
Relative Motion (Standard Joystick) or Change in Motion (Only move based on force in a direction)
D-Pad or Hat Motion (Extremes only and would act like a D-Pad)

Ultimate goal would be to be able to predict motion, based on previous interactions and allow the user to only need to specify changes or other motions.

I also connected an I2C Thumbstick joystick to experiment with the customisations and prove the ability to add unique customisations. The code detects the thumbstick and includes it as additional axis. The thumbstick button is also added.

Next was a simple interface to change the customisations. I opted to switch to the ESP32S3 as this allows for a HTML interface to be set up.

I set the ESP32S3 into AP mode, so the device can be connected to anyware and reprogrammed.

The connection credential are:
SSID: MC
Password: 'helpmemove'

Then you just need to browse to 192.168.4.1:5000, as shown on the display:

1 / 2

As suggested by the contest master for the BUILD2GETHER 2.0 competition, I also worked on designs for mounting the devices to a wheelchair and attachments for limbs, with the ability to got between.

1 / 3

I also experimented with the XIAO ESP32S3 + Grove Shield because you can add lots of options with the big shield. The onboard camera could also be used for face or object tracking, as an added function.

Conclusion

Any of these devices could be used to interact with smart home devices or other helpful things, like adjusting settings on wheelchairs. The code has been written in CircuitPython, so lots of libraries are available. This device could also be paired with a Raspberry Pi, running NodeRed or openHAB to trigger any actions imaginable!

Schematics

Code

# SPDX-FileCopyrightText: 2024 Michał Pokusa
#
# SPDX-License-Identifier: Unlicense

import socketpool
import wifi
import board
import time
import busio
from adafruit_bus_device.i2c_device import I2CDevice
from mpu6886 import MPU6886
import struct
import busio  # type: ignore (This too!)
from joystick_xl.inputs import Axis, Button, Hat
from joystick_xl.joystick import Joystick
from joystick import JoystickUnit

from adafruit_httpserver import Server, Route, as_route, REQUEST_HANDLED_RESPONSE_SENT, Request, Response, GET, POST

AP_SSID = "MC"
AP_PASSWORD = "helpmemove"

print("Creating access point...")
wifi.radio.start_ap(ssid=AP_SSID, password=AP_PASSWORD)
print(f"Created access point {AP_SSID}")

pool = socketpool.SocketPool(wifi.radio)
server = Server(pool, "/static", debug=True)

sampling_ms = 150
minChange = 5

# Axis configuration constants
AXIS_DB = 5
AXIS_MIN = -255  # Minimum raw axis value.
AXIS_MAX = 255 # Maximum raw axis value.
factor = 25

# Axis configuration constants
THUMB_MIN = -128  # Minimum raw axis value.
THUMB_MAX = 128 # Maximum raw axis value.
THUMB_DB = 0 # Deadband to apply to axis center points.

# IMU6866 define
MPU6886_ADDRESS=0x68

i2c = busio.I2C(board.IMU_SCL, board.IMU_SDA)
i2c2 = busio.I2C(board.D1, board.D2) //An I2C device must be plugged in or pullups added
imu = I2CDevice(i2c, MPU6886_ADDRESS)
lcd = board.DISPLAY
jhat = ""
assert i2c2.try_lock()
if 0x52 in i2c2.scan():
    i2c2.unlock()
    jhat = JoystickUnit(i2c2, 0x52)
    jhat.swap_x(1)

sensor = MPU6886(i2c)

hotas = Joystick()

hotas.add_input(
    Button(),
    Axis(deadband=125, min=-255, max=255),
    Axis(deadband=125, min=-255, max=255),
    Axis(deadband=THUMB_DB, min=THUMB_MIN, max=THUMB_MAX),
    Axis(deadband=THUMB_DB, min=THUMB_MIN, max=THUMB_MAX),
)


FORM_HTML_TEMPLATE = """
<html lang="en">
    <head>
        <title>Form with {enctype} enctype</title>
    </head>
    <body>
        <h2>Please adjust the Joystick Settings</h2>
        <form action="/form" method="post" enctype="{enctype}">
            <p>Sample Rate (ms): <span id="sV">{SampleVal}</span> </p>
            <input type="range" min="10" max="500" value="{SampleVal}" id="Sample" name="Sample">
            <p>Minimum Range:<span>{MinValDisp}</span></p>
            <input type="range" min="-255" max="255" value="{MinValDisp}" id="MinVal" name="MinVal">
            <p>Maximum Range:<span>{MaxValDisp}</span></p>
            <input type="range" min="-255" max="255" value="{MaxValDisp}" id="MaxVal" name="MaxVal">
            <p>Minimum Motion Change:<span id="ra">{ChangeVal}</span></p>
            <input type="range" min="1" max="50" value="{ChangeVal}" id="Change" name="Change">
            <p>Set Deadband:<span id="dB">{DeadbandVal}</span></p>
            <input type="range" min="0" max="100" value="{DeadbandVal}" id="Deadband" name="Deadband">
            <p>Motion Options</p>
            <input type="radio" name="Motion" id="RelativeMotion" value="RelativeMotion" checked="checked"> Relative Motion
            <input type="radio" name="Motion" id="ChangeInMotion" value="ChangeInMotion"> Change in Motion<br>
            <br><input type="submit" value="Submit">
        </form>
        {submitted_value}
    </body>
</html>
"""


@server.route("/", [GET, POST])
def form(request: Request):
    """
    Serve a form with the given enctype, and display back the submitted value.
    """
    global sampling_ms
    global AXIS_MIN # Minimum raw axis value.
    global AXIS_MAX # Maximum raw axis value.
    global AXIS_DB # Deadband Change minimum.
    global minChange
    enctype = request.query_params.get("enctype", "text/plain")

    if request.method == POST:
        sampling_ms = int(request.form_data.get("Sample"))
        AXIS_MIN = int(request.form_data.get("MinVal"))
        AXIS_MAX = int(request.form_data.get("MaxVal"))
        minChange = int(request.form_data.get("Change"))
        AXIS_DB = int(request.form_data.get("Deadband"))
        posted_Motion = request.form_data.get("Motion")

    return Response(
        request,
        FORM_HTML_TEMPLATE.format(
            enctype=enctype,
            submitted_value=(
                "<h3>Updated</h3>"
                # f"<h3>Submitted form Sample: {sampling_ms}ms</h3>\n<h3>Submitted form Range: {minChange}</h3>\n<h3>Submitted form Deadband: {posted_Deadband}</h3>\n<h3>Submitted form motion: {posted_Motion}</h3>"
                if request.method == POST
                else ""
            ),
            MinValDisp=(
                f"{AXIS_MIN}"
                if request.method == POST
                else f"{AXIS_MIN}"
            ),
            MaxValDisp=(
                f"{AXIS_MAX}"
                if request.method == POST
                else f"{AXIS_MAX}"
            ),
            SampleVal=(
                f"{sampling_ms}"
                if request.method == POST
                else f"{sampling_ms}"
            ),
            ChangeVal=(
                f"{minChange}"
                if request.method == POST
                else f"{minChange}"
            ),
            DeadbandVal=(
                f"{AXIS_DB}"
                if request.method == POST
                else f"{AXIS_DB}"
            ),
        ),
        content_type="text/html",
    )


server.start(str(wifi.radio.ipv4_address_ap))

while True:

    try:
        gyro_array = sensor.gyro()
        accel_array = sensor.acceleration()
        DB = AXIS_MIN + (AXIS_MAX - AXIS_MIN)/2
        X = int(accel_array[0]*factor)
        if X > AXIS_MIN and X < AXIS_MAX:
            if X > DB-AXIS_DB and X < DB+AXIS_DB:
                hotas.axis[0].source_value = 125
            # print(f"X: {X}")
            elif X < hotas.axis[0].source_value - minChange or X > hotas.axis[0].source_value + minChange:
                hotas.axis[0].source_value = X
        Y = 0-int(accel_array[1]*factor)
        if Y > AXIS_MIN and Y < AXIS_MAX:
            if Y > DB-AXIS_DB and Y < DB+AXIS_DB:
                hotas.axis[1].source_value = 125
            elif Y < hotas.axis[1].source_value - minChange or Y > hotas.axis[1].source_value + minChange:
                hotas.axis[1].source_value = Y
        if jhat:
            X = jhat.get_x()
            if X > AXIS_MIN/2 and X < AXIS_MAX/2:
                if X > DB/2-AXIS_DB and X < DB/2+AXIS_DB:
                    hotas.axis[2].source_value = 0
                    # print(f"X: {X}")
                elif X < hotas.axis[2].source_value - minChange or X > hotas.axis[2].source_value + minChange:
                    hotas.axis[2].source_value = X
            Y = jhat.get_y()
            if Y > AXIS_MIN/2 and Y < AXIS_MAX/2:
                if Y > DB/2-AXIS_DB and Y < DB/2+AXIS_DB:
                    hotas.axis[3].source_value = 0
                    # print(f"X: {X}")
                elif Y < hotas.axis[3].source_value - minChange or Y > hotas.axis[3].source_value + minChange:
                    hotas.axis[3].source_value = Y
                hotas.button[0].source_value = 1 - jhat.get_button_status()
        hotas.update()
        # print(jhat.get_x())
        # print(accel_array)

                # Process any waiting requests
        pool_result = server.poll()

        if pool_result == REQUEST_HANDLED_RESPONSE_SENT:
            # Do something only after handling a request
            pass

        time.sleep(sampling_ms/1000)

        # If you want you can stop the server by calling server.stop() anywhere in your code
    except OSError as error:
        print(error)
        continue

Credits

David Barrett

4 projects • 3 followers

I have a Bachelor of Technology (Computer Technology) from Latrobe University and I keep myself up to date with the latest technology.

Comments

Awards

Honorable Mention Mobility Impairments

Build2gether 2.0 — Inclusive Innovation Challenge

Mobility Control Assistant