Snips Assistant for Emergencies (SAFE)

Overview

Snips Assistant For Emergencies will help you calling to preset contacts and eventually raising a sound alarm.

It can be activated by voice or pressing the button.

You can also set up a satellite pendant for mobility inside your home.

The system has these fundamental elements:

• 1. Main unit or Base.
• 2. Emergency wireless pendant.
• 3. Home router with Wi-Fi and internet connection.
• 4. VoIP account.

This project is about the two first elements, and the other two are assumed.

Main unit or Base preparation

Step1: The Voice Interaction Base Kit and other hardware

The Snips+Seed voice interaction base kit is assembled as the vendor's instructions but dismissing the temperature-humidity sensor. An alarm siren will take its place.

The alarm has to be fixed to the back panel, using the spare mounting spacers and screws.

Then you cut an usb cable, taking the usb-A male part and use the usb powering wires for the alarm (usually red for 5V and black for ground) discarding the data ones. If there is no red and black wires inside the usb cable, you can connect it to your pc and do a test with a multimeter.

Take care of connecting the 5V wire through the relay, so the alarm can be programmatically raised and cleared. Although the alarm sounder is a 12V one, it works fine with less voltage, the noise is lower but still pretty effective.

The usb powering the alarm has to be plug into any raspberry pi B+ usb port.

So now the hardware is fully mounted.

Then it is time to make it more aesthetic by means of 3D printing and assembling some embellisher parts, one protective casing in ABS or PLA and one button cover in flexible material.

And the base module hardware is complete now.

It is recommended powering the SAFE base and the home wifi router through an UPS device (this is a suitable example) to keep working on blackouts.

Step2: SAFE base preparation

The SD card in the kit is preinstalled with snips and some demo apps. You can get the description of the kit and some first setup hints at https://docs.snips.ai/the-maker-kit/dev-kit.

Parameters for Voice Interaction Base Kit
-----------------------------------------
hostname: snips-base
user: pi
password: raspberry
site id: safebase
spi: enabled
i2c: enabled
ssh: enabled

So you should only deploy the SAFE app and the softphone software.

Once you have access to the base raspberry and the wifi is active, there is some preparatory configuration to do.

Set up a fixed ip address, what will ease further steps:

$ sudo vi /etc/dhcpcd.conf

	# Append at the end
interface wlan0
static ip_address=192.168.1.50/24
static routers=192.168.1.1
static domain_name_servers=192.168.1.1

Configure audio to allow microphone and speaker sharing resources with the softphone:

$ sudo vi /etc/asound.conf

pcm.!default {
   type asym
   capture.pcm "sharedmic"
   playback.pcm "speaker"
}
pcm.sharedmic {
   type plug
   slave.pcm "multiapps"
}
pcm.multiapps {
   type dsnoop
   slave.pcm "hw:1,0"
   ipc_key 777777
   ipc_perm 0677
}
pcm.speaker{
   type plug
   slave.pcm "dmixed"
}
pcm.dmixed {
   type dmix
   ipc_key 888888
   ipc_key_add_uid false
   ipc_perm 0677
   slave {
       pcm "hw:1,0"
       period_time 0
       period_size 1024
       buffer_size 8192
       rate 44100
   }
   bindings {
       0 0
       1 1
   }
}

Change volume settings if the speaker is not loud enough:

$ alsamixer && sudo alsactl store

Now it is time to update SNIPS software to get latest features and deploying MQTT tool:

$ sudo apt-get update && sudo apt-get upgrade snips-platform-voice
$ sudo usermod -a -G snips-skills-admin
$ sudo apt-get install mosquitto-clients

The demo assistant has to be removed, as well as its skills, except for snisp-skill-respeaker that shows a beautiful led performance when the assistant is talking or sleeping.

$ sudo rm -rf /usr/share/snips/assistant
$ cd /var/lib/snips/skills
$ sudo rm -fR snips-app-relay-switch snips-app-sht31 

If you are not using the kit but building your own hardware, you can get the skill for leds animation from here https://github.com/snipsco/snips-skill-respeaker.

Finally change the network name:

And reboot.

$ sudo reboot

Step3: SAFE base assistant

To deploy and configure the Snips assistant with the SAFE app in the base device, see the detailed instructions at the SAFE app repository.

You are kindly invited to customize a client name and default contact, and to record a S.O.S. message with your voice, that will be played automatically in an emergency call - automatic call - to the default contact.

Softphone: linphone

The solution in the base and the satellite requires a softphone installed, I have choosen the open source Linphone.

It is required to deploy the console softphone (linphonec) in base and satellite devices, so read carefully and implement the instructions at the resources repository.

You will need a VoIP account to configure the phone. If you have a fiber contract at home, you have many possibilities of configuring VoIP calls, check with your telecommunications operator such option. Otherwise you can create a free account at some internet pages, or at least a temporary or testing account.

Satellite preparation

The pendant is a wireless device based in Raspberry Pi Zero W development board, that is able to speak to the user and initiate a assistance session pushing its button.

It is a satellite device of Snips platform, just with audio capabilities and no installed skills.

The hotword "hey Snips" is toggled off in the satellite because of power saving considerations (wifi off), and the button is the only way to start the assistance. Once the assistance session is initiated, the pendant has voice capabilities and is able to make calls as well, with the same functionalities than the base.

You may prepare and attach more than one satellite to the same base, just beware of changing the id safependant0 for any other id (e.g. safependant1, safependant2...) throughout the configuration.

Every pendant may be customized with a different client name and a S.O.S. wav message, but the contacts list and the default contact (Emergency) are common to all the devices, and configured in the base.

Step1: Create a breadboard hardware prototype

The pendant (satellite) has a bunch of components you have to get.

It is recommendable to wire the elements as stated in the circuit schematic using a breadboard, and test it with the software (next steps) before continue with the packaging.

The connections with RPiZero GPIO are:

• Button one side pin to BCM 17 (pin 11)
• LED anode to BCM 22 (pin 13)
• Button other side pin and LED cathode to resistor to Pi GND (pin 20)
• PB100c 5V to Pi 5V (pin 2)
• Mic 3V to Pi 3.3v (pin 17)
• Mic DOUT to BCM 20 (pin 38)
• Amp Vin to Pi 5V (pin 4)
• Amp DIN to BCM 21 (pin 40)
• Mic and Amp BCLK to BCM 18 (pin 12)
• Mic and Amp LRCL to BCM 19 (pin 35)
• PB 1000c and Mic and Amp GND to Pi GND (pin 20)

To minimize the size and make it usable, the Power Boost 1000c that is the device load-share battery charger and power supply, needs some preparation:

• do not mount USB plug
• remove the JST battery plug
• attach a SPDT slide switch, because once the battery is plugged and has some charge, the device will be "always on" and sometimes it is safe to handle it off (see the vendor assembly recommendations)
• if you want to be more energy efficient, remove the LEDs except the ones for the low battery indicator and the charging indicator

You can read carefully the vendor instructions.

Step2: O.S. and initial setup

Raspbian is the Linux distribution of choice running on the Raspberry Pi. In this guide, we will be using the Lite version. To download and burn Raspbian to the SD card, follow the instructions:

https://downloads.raspberrypi.org/raspbian_lite_latest

https://www.raspberrypi.org/documentation/installation/installing-images/README.md

For this tutorial, we will assume the Raspbian Stretch version 4.14.98 or later.

Once the SD card is ready, there are different ways to do a headless init, but I will describe the setup powering with a HDMI screen and a dongle keyboard attached.

The target configuration is:

Parameters for Voice Interaction Satellite Kit
----------------------------------------------
hostname: snips-sat
user: pi
password: raspberry
site id: safependant0

Do a login with the pi user and configure the hostname and the services we need:

$ sudo raspi-config

Network Options -> Hostname -> snips-sat
Network Options -> Wi-Fi -> (select country) -> SSID -> PASSWORD -> finish
Interfacing Options -> SSH -> enabled

Set up a fixed ip address, what will ease further steps:

$ sudo vi /etc/dhcpcd.conf

	# Append at the end
interface wlan0
static ip_address=192.168.1.51/24
static routers=192.168.1.1
static domain_name_servers=192.168.1.1

Set up wifi if you don't what to plug an ethernet cable.

$ vi wpa_supplicant.conf

# Add wifi network
network={
       ssid="YOUR_SID"
       psk="YOUR_PASSWORD"
 }

And reboot.

$ sudo reboot

Now the Raspberry Pi Zero W is ready for remote ssh access:

$ ssh pi@192.168.1.51

Step3: Audio drivers, codecs and configuration

The custom audio card solution I've chosen:

• I2S I2S MEMS Microphone Breakout - SPH0645LM4H
• I2S 3W Class D Amplifier Breakout - MAX98357A

requires some codecs and drivers compilation.

Maybe it is a better option get an USB card or a hat (as the reSpeaker2mic one), but the power usage will be higher and the final hardware mounting less compact.

First of all some preliminary steps for compilation:

$ sudo apt-get update
$ sudo apt-get install -y git wget curl subversion
$ sudo apt install raspberrypi-kernel-headers

Then the microphone codec compilation and installation:

$ mkdir ics43432
$ cd ics43432
$ wget https://raw.githubusercontent.com/raspberrypi/linux/rpi-$(uname -r | cut -d'.' -f1,2).y/sound/soc/codecs/ics43432.c
$ echo 'obj-m := 'ics43432'.o' > Makefile
$ make -C /lib/modules/$(uname -r)/build M=$(pwd) modules
$ sudo insmod ics43432.ko

To test if the codec is ok, use this command:

$ lsmod | grep ics43432

Now you will configure an overlay to include the amplifier breakout and the mic in a custom sound card that we will call sndrpisimplecar.

$ vi i2s-soundcard-overlay.dts

/dts-v1/;
/plugin/;
/ {
   compatible = "brcm,bcm2708";
   fragment@0 {
       target = <&i2s>;
       __overlay__ {
           status = "okay";
       };
   };
   fragment@1 {
       target-path = "/";
       __overlay__ {
           codec_mic: card-codec {
               #sound-dai-cells = <0>;
               compatible = "invensense,ics43432";
               status = "okay";
           };
           codec_amp: pcm5102a-codec {
               #sound-dai-cells = <0>;
               compatible = "ti,pcm5102a";
               status = "okay";
           };
       };
   };
   fragment@2 {
       target = <&sound>;
       master_overlay: __dormant__ {
           compatible = "simple-audio-card";
           simple-audio-card,format = "i2s";
           simple-audio-card,name = "soundcard";
           simple-audio-card,bitclock-master = <&dailink0_master>;
           simple-audio-card,frame-master = <&dailink0_master>;
           status = "okay";
           simple-audio-card,cpu {
               sound-dai = <&i2s>;
           };
           dailink0_master: simple-audio-card,codec {
               sound-dai = <&codec_mic>;
           };
       };
   };
   fragment@3 {
       target = <&sound>;
       slave_overlay: __overlay__ {
               compatible = "simple-audio-card";
               simple-audio-card,format = "i2s";
               simple-audio-card,name = "soundcard";
               status = "okay";
               simple-audio-card,dai-link@0 {
                   reg = <0>;
                   format = "i2s";
                   cpu {
                       sound-dai = <&i2s>;
                   };
                   codec {
                       sound-dai = <&codec_mic>;
                   };
               };
               simple-audio-card,dai-link@1 {
                   reg = <1>;
                   format = "i2s";
                   cpu {
                       sound-dai = <&i2s>;
                   };
                   codec {
                       sound-dai = <&codec_amp>;
                   };
               };
       };
   };
   __overrides__ {
       alsaname = <&master_overlay>,"simple-audio-card,name",
                   <&slave_overlay>,"simple-audio-card,name";
       master = <0>,"=2!3";
   };
};

The overlay compilation and installation:

$ dtc -@ -I dts -O dtb -o i2s-soundcard.dtbo i2s-soundcard-overlay.dts
$ sudo cp i2s-soundcard.dtbo /boot/overlays

Some configuration:

$ sudo vi /etc/modules

#snd-bcm2835 # comment if exists  

$ sudo vi /boot/config.txt

dtparam=i2s=on # uncomment
#dtparam=audio=on # comment
dtdebug=1 # add for debugging at boot dmesg
dtoverlay=i2s-soundcard,alsaname=sndrpisimplecar # add

And reboot.

$ sudo reboot

If the procedure is ok, the new hardware should be listed with its name in the commands:

$ arecord -l

**** List of CAPTURE Hardware Devices ****
card 0: sndrpisimplecar [sndrpisimplecar], device 1: bcm2835-i2s-ics43432-hifi ics43432-hifi-1 []
Subdevices: 1/1
Subdevice #0: subdevice #0

$ aplay -l

**** List of PLAYBACK Hardware Devices ****
card 0: sndrpisimplecar [sndrpisimplecar], device 0: bcm2835-i2s-pcm5102a-hifi pcm5102a-hifi-0 []
 Subdevices: 1/1
 Subdevice #0: subdevice #0

And you can perform a basic recording and playing test:

$ arecord -D hw:0,1 -c2 -r 48000 -f S32_LE -t wav -V mono -d 3 -v test.wav
$ aplay -D hw:0,0 test.wav
$ speaker-test -D hw:0,0 -c2 -l3 -F S32_LE --test=wav -w /usr/share/sounds/alsa/Front_Center.wav

It is recommended to use a bigger speaker for the tests with more supported power, not the final mini speaker, because if the volume is too high it could be damaged.

Now the ALSA configuration to make the sound card more easily available for the Snips and Linphone software and allow them to share the resources.

$ sudo vi /etc/asound.conf

pcm.!default {
   type asym
   capture.pcm "sharedmic_sv"
   playback.pcm "speaker_sv"
}
pcm.sharedmic_sv {
   type softvol
   slave.pcm "sharedmic"
   control {
	name "Boost Capture Volume"
	card sndrpisimplecar
   }
   min_dB -2.0
   max_dB 20.0
}
pcm.sharedmic {
   type plug
   slave.pcm "dsnooped"
}
pcm.dsnooped {
   type dsnoop
   ipc_key 777777
   ipc_perm 0677
   slave {
       pcm "hw:0,1"
       channels 2
       format S32_LE
   }
}
pcm.speaker_sv {
   type softvol
   slave.pcm "speaker"
   control.name "Speaker Volume"
   control.card 0
}
pcm.speaker{
   type plug
   slave.pcm "dmixed"
}
pcm.dmixed {
  type dmix
  ipc_key 888888
  ipc_perm 0677
  #ipc_key_add_uid false
  slave {
    pcm "hw:0,0"
    period_time 0
    period_size 1024
    buffer_size 8192
    rate 44100
    channels 2
  }
  bindings {
       0 0
       1 1
  }
}

And another reboot.

$ sudo reboot

Now it is time to do a final testing of the default sound devices, and check that everything is ok:

$ arecord -D default -c2 -r 48000 -f S32_LE -t wav -V mono -v test.wav
$ speaker-test -c2 -l3 --test=wav -w /usr/share/sounds/alsa/Front_Center.wav
$ aplay default test.wav

The volume should be regulated, maximum for Boost (mic) and 90% for Speaker:

$ alsamixer && sudo alsactl store

By pressing F5 you display all the controls.

Step4: Custom software to manage the pendant satellite (and Snips partial platform deployment)

The SAFE satellite will not run a Snips skill or app, but only a custom software that interacts with a partial deployment (only audio server service) of Snips platform.

See the detailed instructions at SAFE satellite repository.

You can customize a different client name for the pendant, and also record a different wav S.O.S. message.

Step5: Power saving and enable boot service

As the pendant is intended to be a wireless battery powered device, the energy saving is very important.

The Raspberry Pi Zero W have a low idle consumption, but it is required to make it even lower, because of the small battery attached.

As stated in the installation instructions, it is recommended to disable unused interfaces and services with the script:

cd snips-satellite-safe
./setup_battery_saving.sh

And when the system is ready and tested, disable ssh:

$ sudo raspi-config

Interfacing Options -> SSH -> disabled

To keep open an alternative communication way for troubleshooting, you can enable the UART service if you have a USB-console cable available:

$ sudo vi /boot/config.txt

   # Serial console
   enable_uart=1

And reboot.

$ sudo reboot

The final step is to activate the service that starts the satellite custom software at boot, so when you switch on the device SAFE process is working.

This service will disable the wi-fi communication (when the software configuration option is power_save=1) so be sure in advance that the software configuration and deployment is totally tested.

$ sudo systemctl enable snips-safependant-server

Step6: Packaging and enclosure

To start packaging the device use the double sided mini proto board (7x3cm) and arrange the power supply, the i2s microphone and the i2s amplifier with their headers on, to avoid short-circuits. The wiring will be done in the back side, and at the end of the wiring, shorten the pins (at least in half) with a side cutter.

Now you have to create a small circuit with the button and the LED and the 680ohms resistor and mount it in a 4x7 pins piece of bakelite one sided perfboard, this way:

Despite of the image, you can use a regular resistor (instead of the surface mounting I get), it also fits.

The button-led circuit and the peripherals will be mounted together using the 3D printed inner part 1 (see attachments):

Then you attach the Raspberry Pi Zero W, soldering the wires direct to the GPIO interface.

BE CAREFUL with the pins numbers and specially with Ground, 5V and 3.3V ones. It is recommended the usage of a multimeter to test the continuity among the ground points and power points, and avoid short-circuits.

The following step is to attach the battery, and now you can power on the device, and make some preliminary testing to assure that everything is in place.

To complete the packaging, 3D print the inner part 2 (see attachments) and carefully fold the Raspberry Pi Zero W and the peripherals back to back with the printed part in the middle, like a sandwich.

Now you get a neat package:

Finally 3D print the enclosure (front and back) and the rubber button, and set up with 2mm screws to get the final pendant satellite wireless device.

For a perfect fit, maybe you will have to slightly file the mounting PCB corners, that are 90º instead of rounded as the Raspberry Pi Zero, using a metal file.

The back piece has a small outgoing to be placed below the micro USB power plug (the power booster one, not the RPi Zero W plug).

Assistant dialogue flow

Whenever the client is in an emergency situation, it will be start the assistant pushing the button (for base and pendant) or saying the hotword "hey Snips" to the base.

Once the client starts an assistance, a dialogue is initiated in order to ask for help to a contact person through a phone call. And eventually raise an alarm sound to alert the neighbors.

If the customer is not able to speak or indicate a contact to call or stating that everything is ok, the system will initiate an automatic emergency procedure:

• call to the default contact
• play the sos wav message when the communication is stablished
• finally raise the sound alarm

The dialogue flow in more detail is this picture:

You can find some demonstration videos in the YouTube channel.

Future work and improvements

This project is the first prototype, and there are some improvements and extra pending tasks, most of them related with the satellite pendant.

Improvements:

• Better audio quality in the pendant, try different configuration to avoid cracks and clips, that appears mainly with Snips (but not with ALSA testing).
• Try different speakers to get louder audio in the pendant.
• A semi-waterproof enclosure design.

Extra features:

• Implement low power sound alert in the pendant using power boost LBO pin.
• Include an accelerometer in the pendant to detect falls.
• Move the pendant from RPi Zero W to a lighter and smaller and less power hungry developement board to create a real pendant.

implies developing an alternative snips-audio-server software for this new board.