SmartSlippers - Movement recogintion for grandparents safety

There are currently 700 million people over the age of 65 in the world. Homes for the elderly are full and there is a shortage of nursing staff. The solution to the problem is to extend time period where they can live at home. Older people tend to rely on help from the loves ones, who unfortunately do not have enough time for the constant supervision.

There are many devices on the market that record movement: smartwatches, bracelets, necklaces,... All of the above are invasive methods. Older people who are not in favor of technology may refuse to use such devices. We were looking for a way to make the smart device as noninvasive and invisible as possible - so that the user is not even aware that he is wearing it.

We got the idea for smart slippers. It seemed ridiculous at first, but when we think about it, we find that people almost always wear footwear. Even demented people wear slippers when they leave the bedroom.

Here we present you SmartSlippers - a wearable device that, based on the recognition of movement on the device in real-time, assesses the status of its user and sends messages about changes to the Android application over Bluetooth. It is easy to use and non-invasive. In our application, we implemented the recognition of walking, running, standing still, walking upstairs, and falling. Additionally, we've added code that counts the steps taken.

Machine learning on the edge

A machine learning model for smart slippers was created using the Edge Impulse platform. An accelerometer on an Arduino Nano 33 BLE Sense was used to detect movement.

We acquired data using an Arduino with Edge Impulse firmware to upload data to the web platform directly. We were capturing 2-second samples and cropped some of them. Totally we captured 14 minutes of 5 different movements

• walk - hoja
• run - tek
• staircase walk - stopnice
• fall - padec
• idle - idle

33 features were generated from the motion data using spectral analysis. Then we were experimenting with neural network architecture until we get satisfactory results. This is the final NN architecture with model results.

And here we have the results of the test data.

The model we downloaded as an Arduino library and installed it.

Arduino Firmware

On the Arduino part, we took as a basis the example (continuous acceleration prediction sketch) from the downloaded library. We then added code that allows us Bluetooth connectivity using ArduinoBLE library. We also added a Bluetooth connection LED indicator.

We added a code for counting steps. It is a very basic step recognition. For every sample we calculate acceleration vector magnitude subtracted with earth gravity. For every 2 second window of samples we threshold calculated values and check the interval since last step. Then we have also to check if step was already recorded because windows with data overlap.

Current prediction is prediction which was predicted at least 7 of 10 last predictions. The only exception there is a fall, where we need at least 2 prediction with fall followed with current prediction of idle.

It's important that you add BLEService and BLEIntCharacteristics above setup function. Example:

BLEService copatiService("eba7805c-b406-11ec-b909-0242ac120002");
BLEIntCharacteristic copatiIdle("f7a9b8d6-b408-11ec-b909-0242ac120002", BLERead | BLENotify);

Then in setup function you need to set Local name, Device name, add all Characteristics and Service.

BLE.setLocalName("SmartSlippers");
BLE.setDeviceName("SmartSlippers");
BLE.setAdvertisedService(copatiService);
copatiService.addCharacteristic(copatiIdle);
BLE.addService(copatiService);
BLE.advertise();

Then you send predictions via BLE in run_inference_background function. Example:

copatiIdle.writeValue(1);

Find out more about Arduino code in our Github repository.

Android Application

We created the Android application in the Android Studio environment using the Kotlin programming language. The appearance of the user interface was first created with the Framer tool. Trust us, it's always better to design your app UI upfront, so you don't need to think about that later when adding functionality to your app.

Bluetooth connectivity in Kotlin can be quite complex, so we've used the example of the Bluetooth project described in the PunchTrough guide as a basis. We updated the code and added additional functionality to the application.

The user first comes to the Login screen, to enter username and password. If we don't have the user account already, we can create one in the Signup activity. We also added the option to reset the password if we forget it.

Here are screenshots of our application:

If you haven't paired SmartSlippers with your account yet, you will then enter the Connection screen to search for SmartSlippers nearby. Here is a pairing process:

1. Boot up your Arduino Nano 33 BLE Sense.

2. Wait until RED LED starts blinking. That means that your slippers are not connected.

3. Click Search for Devices button.

4. Choose your SmartSlippers from list of devices found and click Connect.

5. LED on the Arduino should turn GREEN (connected) and you come to the Home screen.

If you already paired to SmartSlippers with your account you will seamlessly connect to your device if it's nearby and come tho the Home screen.

On the Home screen, you can see the large green card if everything is OK (no fall). If a fall occurs, the red FALL DETECTED card is displayed on the screen, we also get an alert and notification. On the home screen, you can also see the current events (walking, running, idle, staircase walk.

To see if your grandma is active enough, the time since the last detected activity is displayed, and you can also see how long the person has been active (walking, running, stairs). Additionally, the number of steps is also displayed. That information is really useful to spot that something is wrong with the person.

Since the movement is recorded even when we leave the home screen, we have transferred the code that calculates the status, times, and steps to three different Services, which are running in the background all the time.

We also created a screen with Settings and a screen with Statistics. We can navigate between screens using the menu at the bottom of the screen.

Find out more about the mobile application in our Github repository.

How it works

3D printed housing

The housing is made so that it can be placed on shoes or slippers with elastics for testing purposes. Because this project is still in the PoC stage, we don't have a battery attached to Arduino. You can use a small power bank, put it in your socks and connect to Arduino with micro USB cable.

Improvements

There is still a lot of room for improvement. For universal use, a machine learning model would need to be improved with more diverse dataset. For longer range data transmission, some form of long-range connectivity would need to be added (e.g. 869MHz, wifi, LORA, or 5G). Currently, the device is powered by a power bank, so it would be necessary to add a small battery.