Interactive ML Walkway Lights

Overview

This project is a set of interactive walkway lights that line the staircase at my house. The lights illuminate around a person as they walk up or down the staircase, fading out behind the person leaving a rainbow trail behind them. The system also uses machine learning to implement speech processing so the system can be controlled via speech commands.

The system uses 12 microcontrollers communicating via UART and I2C, 80 infrared emitter/receiver pairs, and 512 WS2812B RGB LEDs.

Applications

Beyond being aesthetically pleasing, this system has multiple real-life applications. The sensing aspect of the lights makes walking up and down the staircase at night safer because an individual's surrounding area can be precisely illuminated. This makes it so the main lights for the stair case can be left off while still allowing someone to see where they are going and avoid tripping. This also helps conserve energy because only a few lights are turned on at a time.

The voice commands could be expanded such that the animations can be customized to each person. For example, if a person is color-blind, they could say their name and switch the animations to only be colors they can see. Other expansions of the voice commands could be to put the system into different modes, such as "party mode" with faster, brighter animations or "movie mode" with dim, slow, and unobtrusive animations.

System Architecture

The top and bottom halves of the staircase operate as individual units and a central hub connects the two. Each half of the staircase is comprised of 5 microcontroller satellite boards and 40 sensors. These boards communicate with each other via UART (with a bit rate of 115200 bits per second) and are connected in series.

There are 3 classes of satellite boards: one to control LEDs on the IR emitter side, one to both gather IR receiver data and control LEDs on the IR receiver side, and one to gather IR receiver data. They are connected in this order: hub -> emitter side LED controller -> primary receiver board that drives LEDs -> remaining receiver boards.

Connecting the boards in series eliminates the possibility of multiple boards' communications colliding on the communication line, as each UART communication line is between just 2 boards. This also allows commands to be sent to only specific boards in the system; for example, animation data only needs to go to the boards that control the LEDs and thus do not need to be transmitted past them to the boards that only collect sensor data.

The machine learning is implemented separately by a TI CC1352P Launchpad that communicates with the central hub via I2C. The audio classification is separate from the rest of the system due to resource limitations. Storing the data for all 512 LEDs takes a significant amount of memory and the hub board needs to constantly communicate with all the satellite boards, so there would be little space left in memory and CPU power to implement the neural network required for audio classification. So, there is a board dedicated solely to audio classification that only sends commands to the hub board when a valid voice command is recognized.

Controlling the LEDs to follow sensed objects is handled by the MCUs that directly control the LEDs, but the animations played while no object is sensed are controlled by the central hub. This is so that the sensor data does not have to be sent back to the central MCU to be processed; instead, the emitter-side MCU only tells the central MCU if an object was sensed or not.

Connecting all the boards in the system only requires 4 wires: 2 for power and 2 for communication (Tx and Rx on the UART channel). This simplifies installation and makes wire management significantly easier.

Below is a diagram of the system architecture:

Sensing

To sense people as they walk up the staircase, there are 80 infrared emitter/receiver pairs lining the staircase. The emitters are on one side of the stairs and are pointed directly at one photodiode (receiver) so someone walking up the staircase will cast a shadow on the receiver. The photodiode detects the decrease in ambient light, and a decrease in sensed IR light above a certain threshold indicates that an object/person is present.

Sensors (photodiodes) are grouped in 10s, with each group of 10 photodiodes feeding into one microcontroller that digitizes the readings. Grouping the sensors this way minimizes the distance the analog signals has to travel and therefore improves their integrity. The microcontroller for each group sends the sensor values as digital signals to the primary receiver board that controls the LEDs for processing. The primary receiver board then determines which sensors are activated and illuminates the corresponding LEDs. The primary receiver board also sends this sensor data to the controller on the emitter side which performs the identical calculations to set its LEDs the exact same way.

Machine Learning

The machine learning used in this project was powered by Edge Impulse. I used the Edge Impulse platform in conjunction with the TI CC1352P Launchpad and CC3200 Audio Boosterpack to both gather data and classify it.

The first step was gathering data to train the model. I gathered data for background noise (including both no speech and random words), the word "top", and the word "bottom". This was intended to train the model to identify when the words "top" or "bottom" were spoken, so the top or bottom half of the staircase could be illuminated. More commands will be integrated in the future, but I started with these as baseline commands. In total I collected 6 minutes and 30 seconds of test data comprised of 3 minutes of noise and 1 minute 45 seconds of samples of the words "top" and "bottom". Gathering more data would help improve the performance of the classification, but I found this to be sufficient for the system to recognize these two words.

I trained the model to accept 500ms audio clips sampled at 16000 Hz. These samples are first processed by an MFCC (Mel Frequency Cepstral Coefficients) block, which is well-suited for human voice. Then, the processed sample is fed into a Keras neural network which classifies the signal into one of the four categories (top, bottom, unknown, or noise).

Once the model was trained, I built the firmware using the Edge Impulse Studio and uploaded it to the launchpad for deployment. It was able to successfully identify the keywords "top" and "bottom", as demonstrated by the video below.

The next step was to integrate this classification into custom firmware, so the CC1352 launchpad can send a signal via I2C to the main hub when the keywords are identified. However, the CC3200SDK that is required to gather data from the audio boosterpack only supports Windows machines, so I was unable to integrate it into the project using my personal Mac laptop. I have code written for running the edge-impulse software and communicating with an external microcontroller via I2C on the launchpad, but I was unable to integrate gathering new audio samples due to the unavailability of the SDK. As soon as an SDK is available for Mac, it should be fairly straightforward to integrate into the firmware I already have written for the launchpad.

Additionally, several voice commands have already been implemented in the main hub firmware. The video below demonstrates the intended functionality for the voice commands. Here, an Arduino Nano controlled by a computer through UART is sending signals to the hub via I2C that would indicate the voice commands "red", "green", "blue", "top", "bottom", "left", "right", and "quit" (which exits voice command mode and resumes normal operation).

Electrical Construction

A major goal of the project was to make the construction modular in order to simplify installation, allow for future improvements, and let parts of the system potentially be implemented in other future projects. This was accomplished by designing three PCBs:

• Sensing boards on which to mount either an IR emitter or IR photodiode. These boards have 4 connectors on either side: two for power and 2 that connect to an I2C port on the satellite boards (see the next bullet point) to future-proof the system if extra sensors are desired.
• Satellite MCU boards for the microcontrollers that process the sensor data and control the LEDs. These boards have the ability to read and process 10 analog inputs, communicate with 2 external boards via UART, provide power to the sensing boards, control the WS2812 LEDs, and communicate using I2C to (unused) ports on the sensing boards. Each board also includes a reset button, power LED, and controllable LED. Some or all of these capabilities can be used depending on where the board is in the system.
• A central hub board that controls all the satellite boards. This board contains a mount for the hub controller development board, 2 UART ports to communicate with the top and bottom halves of the staircase, and an I2C port to communicate with the machine learning board. Note that there were some design changes made after the fabrication of the hub boards due to UART issues and system architecture alterations, so some post-production modifications had to be made.

Below is the main hub board. Power connectors (one input, one to top half, one to bottom half) are outlined in red, UART connectors are circled in white, and the I2C port is circled in yellow.

Below is the LED controller on the emitter side. The two UART ports are circled in white, the LED connector is circled in green, and the two power connectors are circled in red. UART A (the right UART port) connects to the hub, and UART B (the left one) connects to the primary receiver board that controls the LEDs on the side with the IR receivers.

A the primary receiver MCU board that also controls the LEDs is below. The UART connectors are circled in white; the right one connects to the emitter-side MCU and the left one connects to the first receiver MCU. The LED connector is circled in white and the power connectors are circled in red. The sensor inputs all connect to a 2x5 female header circled in yellow.

Below is a board that only collects and processed receiver data. It is the same as the primary receiver board except that it does not have an LED connector. The color coding is the same as the previous image.

Lastly, an example of one of the emitter/receiver boards is below. An IR emitter and/or receiver can be mounted on this board to either produce or sense IR light, respectively. The outline of a single board is highlighted in white, and the 2x2 connectors that join these boards to neighboring boards are outlined in red. This particular board is a receiver, and the IR photodiode is circled in yellow. Note that the unused I2C connection is below the photodiode.

Physical Construction

The sensors and LEDs are enclosed in a wooden housing that keeps all the electronics secure and blends the system into the staircase. The housings are an L shape, with the sensing circuitry sitting vertically and the LEDs facing down.

Each section of the staircase corresponds to one housing, and all boards within that housing can be mounted as one unit. To connect the units together, only 4 wires are needed: 2 for power that connect to screw terminals and 2 for communication that connect through a 2-pin JST header. Because there are only 4 wires, they can be easily hidden so the housings are barely noticeable on the staircase.

Future Improvements

The simplest addition to this project is to add more animations. This is the easiest update to make since it only requires updates to the hub MCU.

Adding more speech commands can increase the versatility of the system. Users could be added to the system so that a person could say their name and have the system alter its behavior. Or, the user could set the system into different modes. These modes could include turning off the sensing, dimming the lights, or making the lights more reactive to changes. The commands "red", "green", "blue", "top", "bottom", "left", "right", and "quit" are already coded into the hub board, but adding more would be very simple to add.

A long-term potential enhancement would be to use the system as a graphic equalizer, taking the sound it observes and displaying the amplitude of different frequency bands on the LED strips. There is already a microphone in the machine learning board, and a voice command could be added to transition the system into graphic equalizer mode in which the audio input is decomposed into different frequency bands. The amplitude of each frequency band could be sent to the hub MCU through the already existing I2C connection which would display this information just like any other animation.

One other addition would be to add environmental sensing to measure ambient light or more. Each of the satellite boards has an unused I2C port that could be connected to various sensors that can read the environment and alter the behavior accordingly. For example, if more ambient infrared IR light is detected, the sensing thresholds could be adjusted accordingly. Or, the colors of the lights could be adjusted based on the ambient temperature.