Published March 10, 2019 © LGPL

PI Vision

Introduction low-cost IoT device for mobility-impaired users.

ExpertFull instructions provided6,518

Things used in this project

Hardware components

Raspberry Pi 3 Model B

For its advanced characteristics and speed, especially for the real-time processing of vision data, the best fitting device is the last Raspberry PI 3B+ model

Generic 5 inches HDMI monitor

Note that the device does not require a monitor. The display is only present in the prototype for testing feedback of the detected face and eyes movements

onsemi Omron Vision face and expressions recognition

The Omron Vision module has the key role to detect the right image and pre-process a set of recognition parameters, then processed, filtered and refined by the Raspberry PI

Arduino MKR1000

This component is the endpoint of the device able to control a generic computer (no driver and not a special computer is needed), tablet or any kind of device emulating mouse and keyboard.

Adafruit NeoPixel Ring: WS2812 5050 RGB LED

In this prototype I have used a NeoPixel 8 LED bar.

Software apps and online services

Raspberry Pi Raspbian

Software developed with C++, Python and other platforms

Arduino IDE

C/C++ software used to interface the Arduino with the Raspberry PI and the controlled computer

Inkscape

Vectorial design software used to create the laser cut components

Hand tools and fabrication machines

Soldering iron (generic)

Laser cutter (generic)

Story

Introduction

In the past I worked more than ten years in the field of artificial imaging, image recognition and participated to several researches, starting by the mid of 1980 when I was living in Italy.

In the second mid of the 1990 decade I had several opportunities to face the challenges related to use the face and eye detection movement as a support for mobility-impaired persons. We know there are degenerative illnesses that progressively reduce the mobility of the patients, until they are constrained to a bed in an almost complete immobility.

As far as I know, nowadays patients can count on devices able to recognise small gestures of the face as well as eye movements. These devices has an incredible high cost, that dramatically reduce the number of people that can access to these resources. Another aspect is that most of them requires special computers or adapted ones, resulting in a certain complexity for usage, installation and setup.

After I have reviewed for Element14 the relatively new component by Omron Vision able to detect a number of face gestures and expressions I decided to see if it was possible to create an easy-to-use, IoT device specifically though for this kind of mobility-impaired users.

The Omron Vision component used to detect the face movements and eyesight direction

The PI Vision Project Assumptions

The key objectives of the PI Vision project can be focused in few words:

Easy to use
No special computer needed, a USB port is sufficient and should work on tablets and smartphones too
Low-cost and extremely affordable (price no higher than $500)
Easy to interface to non-computer devices
Networked
Open source

The PI Vision Motion Control Approach

The recognition approach of the device starts from the data that can be acquired by the Omron Vision component. These are not final data but a series of 3D spatial information that gives precise knowledge along the time line of the user small face gestures and eyes orientation.

The Raspberry PI of the prototype. Note that the HDMI 5 inches monitor is used only for testing feedback. The usable device is a headless unit and and external monitor should be used only for the initial calibration

This step of the recognition is also the heavier part in terms of processor usage, but the Omron component can provide updates almost fast that the final data processing can be took in charge by the Raspberry PI 3B+ in a reasonable amount of time.

The resulting data are converted in simple control codes interpreted by the Arduino MKR1000.

The Arduino MKR1000 in its prototype board

The Arduino board, as the endpoint of the PI Vision device, covers two fundamental roles:

Convert the control codes to emulated mouse movements and keyboard keypresses
Generate a feedback on a small RGB NeoPixel LED bar to acknowledge the user what is the response currently in action.

As you can see in the video above the LED strip has an intuitive color coding sowing then the mouse is moving, double click selected and other features, as well as a different color when a key is pressed.

In the PI Vision prototype the LED strip is boxed together with the Arduino MKR1000 but in a production version the strip can be attached on the monitor frame easy to view without disturbing the usage.

The Test Platform

The test platform positioned together with a small laptop. Back view

Thinking to the most practical usage, the Arduino-based device is connected to the computer via USB that also power it. This means that no wiring is needed to connect the end point directly to the Raspberry PI and Omron Vision: two separate boxes connected via WiFi.

The test platform viewed by the user side.

Most of the commercial devices I saw position the camera on the top side of the monitor frame. This limits the usage to a specific device and create some complications in positioning the stuff respect the point of view of the user.

This kind of solution has the advantage to freely position the monitor (or any other display device) independently by the camera sensor that should be oriented correctly for detection, to grant the better mobility-impaired user experience.

Networking

There is one more advantage to connect the Arduino MKR1000 endpoint via WiFi to the sensor-tracking unit than the better positioning. With this configuration the Raspberry PI B3+ - taking the advantage it has both Wifi and Ethernet available together - act as the system access point as well as acting as a router-proxy between the Ethernet and WiFi.