Senior Guardian Frame

Introduction

Fall has become a major cause of injury for senior citizens, it has become the most frequent cause of death for those who are age 65 and above.

High-tech adoption for senior citizens seem to be really lacking, as many products are created at hackathons but very few makes it to the market. And tech products targets senior citizens tends to be expensive due to use cases and market availability.

Luckily, more and more technologies such as IoT, Cameras are coming with SDK that makes those items programmable. Although these items that's not originally targeting senior citizens, through software SDK we can easily unleash innovation and make specific use cases for senior citizens.

According to studies in the Netherlands, more than 90% of the people wear glasses after age 50 and almost everyone needs some form of visual aid by age 75. Since majority will have a glass frame at that point, building fall detection on the glass itself significantly helps the elderly since adoption is already in place.

Bose Frames

Bose frames are one of those products, the frames are built for Bose AR and spatial sound, it comes with Miniaturized Bose Speakers, Metal Hinges, Bluetooth connection, compass, accelerometer, gyro-scope, microphone and tap sensors, it is Shatter and Scratch resistant. The battery life is 3.5 hours of music listening, but if we use it as a glass it would last much longer.

The glass was originally made for early adopter market, luckily Bose opened up their Bose AR SDK to try to inspire innovation. As eye sight deteriorate with age, glass has become an essential component for most senior citizens. Since we have access to gyro scope and tap sensor, these would allow us to build out an necessary components that can be used to help senior citizens.

In this app, we will use the gyro scope to detect the sudden fall, use the double tap for emergency call, voice button to active google assistant in order to control smart home, and pill reminder to help users to remind their pill time.

Step 1: Changing into prescription or custom glass

The Bose frame comes with default scratch resistance sunglass, but it can easily be changed into prescription glass, or nothing at all for those who does not need prescription glasses at that age. For this project I have taken out the sun glasses, but switching lenses can easily be done through this video.

Step 2: Gather material requirements and download SDK

We first build out our app by getting Bose Frame and Android phone, take out the glasses as shown above video, and connect them via bluetooth. Upon connecting to Bluetooth you should have following screen.

You can download the app from https://play.google.com/store/apps/details?id=com.bose.monet in order to make sure the glass is correctly setup and connected.

After the connection we can download Bose AR SDK from

https://developer.bose.com/bose-ar/bose-ar-downloads

In this article we will focus on Android, we can start with data-example as a base to build out our app.

Step 3: Fall detection algorithm

The fall detection is based upon the algorithm number 2 described in "Comparison of low-complexity fall detection algorithms for body attached accelerometers" authored by Maarit Kangas, Antti Konttila, Per Lindgren, Ilkka Winblad, Timo Jamsa and published in Gait & Posture 28 (2008) by Elsevier (search for the paper here)

Detecting fall can be seen through the following code using accelerometer

double threshold = (fallDetected == true) ? fallenThreshold : normalThreshold;
mGravity = values;
mAccelLast = mAccelCurrent;
mAccelCurrent = (float) Math.sqrt(mGravity[0] * mGravity[0] + mGravity[1] * mGravity[1] + mGravity[2] * mGravity[2]);
float delta = mAccelCurrent - mAccelLast;
mAccel = Math.abs(mAccel * 0.9f) + delta;
if (mAccel > maxAccelSeen) {
    maxAccelSeen = mAccel;
    //string message = "Increased";
}


Log.d("sensor", "Sensor ServiceX: onChange mAccel=" + mAccel + " maxAccelSeen=" + maxAccelSeen + " threshold=" + threshold);
if (mAccel > threshold) {
    Log.e(sensorValue.sensorType().toString(), String.valueOf(values[0]) + "," + String.valueOf(values[1]) + "," + String.valueOf(values[2]));
    maxAccelSeen = 0;
    if ((fallDetected == true) && (mAccel > fallenThreshold)) {
        Log.e("fall", "fall detected");
    } else {
        if ((fallDetected == false) && (mAccel > normalThreshold)) {
            fallDetected = true;
            Log.e("fall", "true fall detected");
            t1.speak("Fall detected, please see if you can move", TextToSpeech.QUEUE_FLUSH, null, "txtFall");
        }
    }
}

Upon falling, we can immediately detect whether the user is moving via accelerometer as well, after a few seconds, we can send a text to their emergency contact.

First this would require SMS permission

<uses-permission android:name="android.permission.SEND_SMS" />

We can detect 8 seconds of inactivity then send the SMS

if (fallDetected)
{
    if(mAccel < 0.1)
    {
        laycount++;
        if(laycount >= 100 && movecount <50)
        {
            //8 seconds
            smsHelp();
        }
    }
    else {
        //reset
        movecount++;
        if(movecount >= 50)
        {
            movecount = 0;
            laycount = 0;
            fallDetected = false;
            t1.speak("It seems you are ok, you can double tap to make emergency phone call", TextToSpeech.QUEUE_FLUSH, null, "txtFall");

        }

    }
}

private void smsHelp()
{
    t1.speak("You are not moving, sending a text to your emergency contact", TextToSpeech.QUEUE_FLUSH, null, "txtFall");
    SharedPreferences preferences = PreferenceManager.getDefaultSharedPreferences(this.getContext());
    String smsNumber = preferences.getString(this.getContext().getString(R.string.contact), "");
    if(!smsNumber.equalsIgnoreCase(""))
    {
        String helpMessage = "I have fallen and need help, sent by Senior Guardian Frames on Android";
        SmsManager smsManager = SmsManager.getDefault();
        smsManager.sendTextMessage(smsNumber,
                null,
                helpMessage,
                null,
                null);
        Toast.makeText(this.getContext(), "Your contact has been notified",
                Toast.LENGTH_LONG).show();
    }
    movecount = 0;
    laycount = 0;
    fallDetected = false;

}

Additionally, we can make sounds via Text to speech, to make sure the person is not lying on the ground.

Step 4: Getting double tap to make quick phone call

Bose Frame also comes with tap sensor, in this step we will focus on the tapping sensor to make it easier for seniors to connect with their emergency phone call, or just to connect with their loved one easily. The tap spot is placed as picture below.

To program the taps, we will first need permission

<uses-permission android:name="android.permission.CALL_PHONE" />

After that we can program the gesture sensor in the same file to make the phone call.

mViewModel.gestureEvents()
        .observe(this, event -> {
            final GestureEvent gestureEvent = event.get();
            if (gestureEvent != null && gestureEvent.gestureData().type().name().equalsIgnoreCase("DOUBLE_TAP")) {
                SharedPreferences preferences = PreferenceManager.getDefaultSharedPreferences(this.getContext());
                String phoneNumber = preferences.getString(this.getContext().getString(R.string.contact), "");
                if(!phoneNumber.equalsIgnoreCase(""))
                {
                    t1.speak("Calling Guardian", TextToSpeech.QUEUE_FLUSH, null, "txtFall");

                    Handler handler = new Handler();
                    handler.postDelayed(new Runnable() {

                        @Override
                        public void run() {
                            Intent intent = new Intent(Intent.ACTION_CALL, Uri.parse("tel:" + phoneNumber));
                            startActivity(intent);
                        }

                    }, 1500); 
                }
            }
            else if(gestureEvent != null && gestureEvent.gestureData().type().name().equalsIgnoreCase("NEGATIVE")) {

            }
        });

Now you can test the app and try to fall to see if it works.

Step 5: Connect with Google Assistant

This step is fairly easy as it's already built in and does not require programming, by click and hold power button the frame would launch google home's assistant, this would allow users to setup reminders for medication, or turning on/off the IoT lights.

Activating connect button can easily launch Google Assistant, making it even easier and accessible.

Step 6: Demo

Here comes the demo of everything we built :)

One More Thing

This article is focused on user wearable that can detect fall, additionally, we can place cameras around and detect falls using AI live video feed to detect falling in addition of wearables. This way we can create redundancy for safety for senior citizens in their home.

We will document how to do that via project link https://www.hackster.io/308965/guardian-camera-272431