Guarding Angel:(Baby, Family in-Car and Home Assistant)

Guarding Angel:(Baby, Family in-Car and Home Assistant-Monitoring Systems)

Being a SensorTiled based mean it’s a virtual assistant but it’s also a physical assistant that is present all the time in real-time.

Reduced baby cot death and fatigue alert, physical health condition, sleep position, breath and anti-choke alarm since most parents sleep separately from babies and babies are always in their cots. Camera monitoring alone is not enough since it difficult to sense breathing, snort, pulse rate, turning position, turning degree, room temperature, humidity and many more around newly born and toddlers. Online and Off-line 24/7 monitoring and alert in real-time. With a specialised Mobile App, a mother can caress baby to sleep, be alert when to check on baby inside the cots and visualised sleep pattern of the baby (daily, week charts for GP and visiting Nurse.

Help expected Mother to perform well during pregnancy. Help monitor breathing, pulses, siting and sleep position. Alert for out of position, resting time, glucose level if possible. Virtual Assistance, Voice based, Gateway hub to Cloud connected mobile-phone to smart devices mean partners and next-of-kin can be as nearer as possible to their loved ones. The App will help informed pregnant mother about their daily routines, patterns and what they should be doing well or stay away from.

In-Car: sense pressure variation due to improperly close car doors, use with toddler car pram for better position and lock. Interface with GPS and OBDII to monitor, alert and as an adviser driving behaviour. for Single parent and large family, vacation, holidays, visit all covered.

There are many other useful application areas, cases and scenario. This will be just a quick prototype implementation that is extend-able to many related areas including wearable, Care Assistant for Elderly People, IoT, Machine learning and many more....

Features:

• Identity

• Motion monitoring

• Position Sensor

• Services

• Simulation

• Realtime Marketing

• Cross language Rest API (Internet, Intranet and Cloud)

• Distributed(offline/online Sync) connected smart devices

• Plugins (extensions and customisation)

• Things Auto-discovery (IoT, GPS, Location Based)

• Security, device movement, burglary detector

Market: Even I am focusing on medical, children, pregnancy, parenting and family. The same prototype design can be customised for vertical and lateral integration to expand into retail, transportation, home security/monitoring, sports and many other sectors/areas.

The edge and the advantage of this particular design is being natural and tag free. Unlike others, toddlers need not to be tagged, there is no need to wear anything or belt a plastic tagging device to toddler's arm, legs, etc... Its wearable and its being part of the clothes, flannel, blanket, bed-sheet, etc... It can be removed and clip back at any time without affecting its operation. Being compact, wearable, waterproof mean that its durable, tough and last longer.

Uses:

Its programmable thorough a drop dead drag & drop, voice enable, intelligent machine learning and wizard assisted mobile application that will accompanying the prototype. The App and the cloud based will provide facilities to perform various visualisations, data analysis, gesture and virtual advice for various conditions and situations that may arise during uses.

User can view how they are performing, parent can view toddler sleeping mode, snort relative to position, pulses relative to activity level to breathing rate, be advise on what to do, how to caress toddler or talk to midwife if needs be.

Steps:

Part-1: Intial Testing of SensorTile Development Kit and Apps

First thing is to test that your SensorTile, the two cradle all working and can communicate with the Mobile Apps. This is important as the first section/phase of the whole systems that will be responsible for collecting data and transmit those data to other areas including the gateway and the cloud.

In this project, I will be working at the source code level and manipulating the Android App on Android Studio, You can use Eclipse, Netbeans or any editor of your choice. Due to time constraint I will only focus on the Android platform but same steps applied to other environment like iOS, Samsung, Intel, Windows, etc...

Below show screenshots and annotated steps taken during the hacking periods.

First download the source code for these sdk(s) from the github portal. Note: You will do use --recursive keyword since most contains submodule. so do the following:

• git submodule add https://github.com/STMicroelectronics-CentralLabs/BlueSTSDK_Android.git BlueSTSDK

• git clone --recursive https://github.com/STMicroelectronics-CentralLabs/STBlueMS_Android

• git clone --recursive https://github.com/STMicroelectronics-CentralLabs/STSensNet_Android

You will need to do a bit of coding to customise the BlueSTSDK. STblueMS ans STSenseNet.

BlueSTSDK is the simplest and the starting point in understanding the protocol, send and receive between SensorTile and the mobile App. A quick run through:

The DevicePreferenceFragment.java enable you to specified where and how store the data are stored in your device(e.g. on phone, sdcard, etc...)

The FeatureListActivity.java lets user select one feature, request to receive it from a selected sensor and display the callback notification.

The ScanActivity.java as from the file name is responsible for scanning for local compatible devices the STSDK supported. The time limit was set to 10sec but if you are on weak network or a bit of distance then you may need to increase the time limit to limit cutout time. It also enables you to register and saved instances of connected node devices. The good news is that the SDK is pretty well documented so if you know a bit of Java and Android environment then it should be possible to customise the SDK for your target user. I will encourage you to go through the SDK code for better undestarding of its usage/adaptations.

Screenshot during initial testing and debugging:

Wont run on my Server Emulator:

I must confess here that I am running on an Ubuntu 16.0 Server, not a desktop which means no graphic card. I may be force to use a desktop or MS Windows 7 :(

I got my Samsung GT tablet connected for debugging so below is the screenshot of the STSDK_Android running and communicating with the SensorTile board.

The STSDK is the base for the two to follow next before we get code underlying those functions for temperature, pressure, magnetometer, gyroscope, voice input and accelerometer. There is also humidity on the mini-cradle but I thought it would have been nice to have all of the sensors on the chip(SOC (sensor on chip)/SOM(sensor on module)). Anyway we come to those part later.

Let us examine the next level of the SDK which is the STBlueMS was the next level up with more features, serves as enhanced version of the STSDK. Its a combined STSDK and SDK_GUI for interactive and bi-direction communication between SensorTile board and your mobile App.

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
 package="com.st.BlueMS">
    <uses-permission android:name="android.permission.BLUETOOTH" />
    <uses-permission android:name="android.permission.BLUETOOTH_ADMIN" />
 <!-- needed for the BlueVoice demos -->
 <uses-permission android:name="android.permission.INTERNET" />
    <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
 <!-- add for paho -->
 <uses-permission android:name="android.permission.WAKE_LOCK" />
 <!-- Memes Sensor Fusion vibrate when a free fall is detected -->
 <uses-permission android:name="android.permission.VIBRATE"/>
    <uses-feature
 android:glEsVersion="0x00020000"
 android:required="true" />

A quick look under the hood as shown above the Android manifest xml with various system access permission for Bluetooth, local-network access, internet access, device wake,vibration/shake and that of openGL graphic needed for operation.

Android is big bloated necessary evil. I worked with many languages from C/C++, Python, Ruby, Go and many more... but I found Java of course Android bloated even beyond .Net framework so we need an Android NDK here to do a transpiler into native code.

The coding are annotated/commented. Under the uses section, you various coding for voice, nfc, temperature and for other sensors on the board.

DemosActivity.java implement fragment for logging, cloud, gesture, proxmity, etc.. see below:

private final static Class<? extends DemoFragment> ALL_DEMOS[] = new Class[]{
 EnvironmentalSensorsFragment.class,
 MemsSensorFusionFragment.class,
 PlotFeatureFragment.class,
 ActivityRecognitionFragment.class,
 CarryPositionFragment.class,
 ProximityGestureRecognitionFragment.class,
 MemsGestureRecognitionFragment.class,
 PedometerFragment.class,
 AccEventFragment.class,
 SwitchFragment.class,
 BlueVoiceFragment.class,
 BeamformingFragment.class,
 SourceLocFragment.class,
 HearRateFragment.class,
 CloudLogFragment.class,
 MotionIntensityFragment.class,
 CompassFragment.class,
 //MultipleLogFragment.class,
 NodeStatusFragment.class,
 //FeatureDebugFragment.class
}; 

I must notify you that the SDK framework is not only for the SensorTile but also cater for other boards like Blue_Coin and NUCLEO family. You can more information at the STMicroElectronics website.

Also OTA (Over The AIR) upgrade facility during a quick tried, at least for my test so I went on eBay to order a NUCLEO board as I do not want brick the SenSorTile using Arduino UNO ( I could not find information on programming the Sensor|Tile using Arduino. Since I have only one of it, unable to buy only the SensorTile module alone).

OTA code extract is as shown below:

/**
 * Create a dialog Fragment that warning the user that is using an old FW
 */
public static class UpdateDialog extends DialogFragment {
 private final static String OTA_APP_ID = "com.st.STBlueDFU";
    private final static String OTA_UPGRADE_ACTION = OTA_APP_ID+".OTA_UPGRADE";
    private final static String OTA_NODE_TAG = "OTA_NODE_TAG";
    private final static String OTA_NODE_NAME = "OTA_NODE_NAME";
    private final static String OTA_NODE_TYPE = "OTA_NODE_TYPE";
 /**
     * return true if there is an app that can handle the intent
     * @param updateIntent intent that will start the dfu app
     * @return true if an app for handle the intent exist
     */
 private boolean hasDfuApp(Intent updateIntent){
        PackageManager packageManager = getActivity().getPackageManager();
        return packageManager.queryIntentActivities(updateIntent,
 PackageManager.MATCH_DEFAULT_ONLY).size()>0;

Work in the code section highlighted below is you want to customise the App for example communication with other devices local or remote, with a remote server or cloud.

For example you want to send data to cloud then look at the IBM Waston coding section 1 of the above screenshot to implement your own (Clone/modify it). Under the Cloud section of the code you will also find code for the generic mqtt that you can study, customise for bringing the SensorTile board to an area of interest e.g. wearable for a gym, bike ride, baby monitoring device (this is part of my project), talking to robots, controlling home devices and so on...

It will be difficult cover every aspect of the SDK code snippet otherwise this will becoming like a manual or text on SensorTile. Below is code extract for the IBM Watson cloud and assigned port for the rest-API

public class IBMWatsonQuickStartFactory implements MqttClientConnectionFactory {
private static final long CLOUD_DATA_NOTIFICATION_PERIOD_MS = 1000;
 //address where send the datas
 private static final String QUICKSTART_URL="ssl://quickstart.messaging.internetofthings.ibmcloud.com:8883";
 //private static final String QUICKSTART_URL="tcp://quickstart.messaging.internetofthings.ibmcloud.com:1883";
    //page where see the data send to the cloud
 private static final String QUICKSTART_PAGE_DATA =
 "https://quickstart.internetofthings.ibmcloud.com/#/device/%s/sensor/";

Study the code for your own adaptation. Other code of interest are the Google ASR section of the code. Look into the section and duplicate or modify for other speech engine e.g. Kaldi engine, CMU Sphinx or whatever suites you.

Snapshot of a running:

The demos apps just shown above demonstrated some of the features of those mobile apps for the SensorTile development kit. STElectronics made few variation of SensorTile in may of their NULCLEO board family.

There are issues during development and implementation cycles but they are what we normally encounter during project phases.

The first part is hooking up the SensorTile cradle with a Nucleo board as a programmer as shown below:

Next I install ST-Link Utility both on my Linux box and a side by Windows 7 machine. See below:

Video demonstrations: https://vimeo.com/user56015444 this is links to various videos that demonstrate many features of SensorTile board, the Apps and data to the Cloud.

The next step is to customise the App, the assets, port the hardware to Python environment so that it can be hooked up to either a local gateway hub or WiFi and then to a cloud. I already tested SensorTile and have it connected to IBM Watson cloud but it will be nice to have a generic dialog to permit people to configure a cloud account.

Project Run-Test : Manipulations of images used in the project for various positions of toddler when alone or in the cot. Android SVG creation was a bit of challenge.

Summary:

• Porting of MicroPyhton will continue even after the competition since my interest as arose to get the porting done.

• Integration with HomeAssistant, Supla-Cloud, Domoticz, Gladys and HomeGenie as started gradually but may take some time but plan is in place in one of community project at London Hackster events.

• SensorTile is still a bit expensive but we hope the price will come.

• It has been a knowledge based exploration and proof of concepts that show that SensorTile can be deloyed successifully in wearable gear from personal assistant, child monitoring, tracking, smart devices, robots and many IoT environments.

• I will push some materials my IoT github but some of the resource used are already attached and made availabled .

Many thanks for the SensorTile Kit and for the opportunity to take part in the competition. I am hoping to explore more avenues with SensorTile and to use it in my teachings.

God blesses!!!

Best regards,

Sanyaade