WalaBreathe - A Wireless Breath To Speech Assistive Device

Introduction

According to the World Health Organisation more than 500,000 people suffer from spinal cord injury each year. It is also noted that and an estimated 20-30% of them show clinically significant signs of depression.

These injuries often disable the motor neuron functions of the people, leaving them in wheelchairs. It also often causes auditory impairment which makes it difficult for them to have day-to-day conversations with their loved one's.

There are several assistive devices that have been built to aid paraplegic users communicate, but often they are bulky, hard to use, slow and expensive costing thousands of dollars. Some of them are :

• BCI (Brain Computer Interface) devices that convert EEG data to Speech

• Cheek sensors (IR Switch) (in case of Prof. Stephen Hawking)

• Camera trackers (Eye)

In the last couple of years, a breath to speech based assistive device called 'Talk' was developed. It took breath input from a MEMS microphone and is a wired device. Talk did show how breath could be an interesting area to explore for the further development of assistive devices.

What is WalaBreathe?

WalaBreathe is a completely wireless breath to speech assistive device that uses Radio Frequency reflectance data from the Walabot Pro to convert a users breath into audible speech.

The WalaBreathe program is written in Python, it uses few external python libraries like:

• Matplotlib Pyplot for plotting breath data in real-time

• E-speak Text-To-Speech engine (alternatives pyttsx or Google TTS)

Materials Required and Hardware Setup

1. Walabot RF based Imaging Sensor (More Info)

The instructions to setup walabot on your PC is available on the walabot page under the Quick Start Guide : https://walabot.com/getting-started .

Software Developement

1. SDK Introduction

The walabot sensor gives a programmer the ability to program/work with the raw signals in various ways.It is surely going to be a super-fun experience programming the walabot sensor and I'm sure it can help shape up your ideas.

The Walabot SDK is available for both Windows and Linux, I worked using the Linux SDK on Lubuntu.

2. Setting up Walabot SDK and Programming using Python

I initially started programming for Walabot using C++ as I was better at C++ than Python, but I hit troubles when it came to plotting graphs in real-time etc.I soon moved to python and found it more comfortable and easy to maintain my workflow in python.

If you don't have the Python IDE installed, Python3 is available at : https://www.python.org/downloads/

Once that's done, you can download the Walabot SDK for your OS. In my case I downloaded the .deb file and installed it via the terminal. Detailed information about the Walabot Python API is available at : https://walabot.com/api/_pythonapi.html

Once you've installed the .deb file you can find various example programs written for both C++ and Python in the source directory. The example programs are also available on the Walabot API website. For this project, I would recommend going through the BreathingExample python program.

3. WalaBreathe Programming Begins!

One of the first things I wanted to do was to visualize the breath magnitude(named 'energy' in the program) in real-time. For doing this I used pyplot which is an open-source python based plotting library.

However, to plot in real-time and visualize it I had to search through various techniques and came across using scatter plotting and dynamically updating the graph using plt.ion() to enable interactive plotting and plt.pause() to draw new data.Below are the main parts of the program explained in detail.

• Part 1: Plotting 'energy' values in real time:

To start using pyplot with walabot you'll have to import the necessary libraries.

from __future__ import print_function  
from sys import platform 
from os import system 
from imp import load_source 
from os.path import join
#pyplot library for plotting breath 'energy' in real-time  
import matplotlib.pyplot as plt 
import numpy as np 
import time 
import array 
import os #for e-speak 

Next, we'll have to define the maximum and minimum values of the 'x' and 'y' axes of for our graph and also enable interactive plotting.

#pyplot function for mentioning minimum and maximum 'x' and 'y' axis values
plt.axis([0, 1000, 0, 100])  
plt.ion() #enables interactive plotting  

In the 2D-plot, the y-axis corresponds to the Breath Magnitude or 'Energy Value'. Therefore, to have a graph that shows the breathing values changing through time, we need to automatically increment the x-axis. For doing this we can use a 'counter' that increments the x-axis by 1. Also, the energy values by default from the example program only gives a very small value. Therefore by multiplying it with 1000 we get values in range 0-100.

   #pyplot function that allows us to scatter plot breathing 'energy' values 
   plt.scatter(counter,energy*1000)    
   plt.pause(0.005)   

That takes care of the real-time plotting of Breathing 'Energy' value.

The results should look something like this:

• Part 2 : Detection of Breathing/Not Breathing

Now that the data in visualized in real-time, we can note many characteristics of the way we breathe. The first thing I wanted to do was to identify when I am breathing and when I am not. On observing the real-time scatter plot of breathing energy values, I noted that magnitude 0-3 meant that there was very less movement or no significant breathing. And values >3 was convincing enough to be taken as 'breathing'. I have two counters, namely the b_counter which stands for 'breathing counter' and the nb_counter which stands for not_breathing counter. They are incremented accordingly to keep track if the user is breathing or not breathing.

   if b_magnitude>3:    #user is breathing 
       b_counter+=1 
   else:    #not breathing 
       nb_counter+=1 

• Part 3 : Classification of long and short breaths

To make a assistive device that converts breath of a user to speech, I had to first identify long and short breaths. The long and short breaths correspond to the 'dash' and 'dot' of morse code. Once the users breath's out his expected command in morse code, we then convert it to English. But, the first step is the classification of long and short breaths.

For this, I started giving a series on long and short breath's and started making observations from the real-time scatter plot.I noticed that long breath's rose to higher magnitudes than short breaths.

On the basis of this, I started recording 15 energy*1000 values at a time and then classify them to long/short breath depending on whether energy values where 'high' or 'low' for most of the time. The 'b_counter' and 'nb_counter' keeps increment depending on if the user is breathing or not until gen_counter==15

#taking 15 'energy' values to classify into 'long' and 'short' breaths 
  if(gen_counter==15):
       if(b_counter>0 and nb_counter>0): 
           if(b_counter>nb_counter):     
#if energy magnitude is 'high' most the time among the 15 readings
#then it is a 'long breath' 
               print('Long Breath') 
               list_counter+=1     
           else:    
#when energy magnitude is 'low' most the time among the 15 readings
#then it is a 'short breath'     
               print('Short Breath') 
       if(b_counter==0):             
           bn_counter+=1         
       gen_counter=0 
       b_counter=0 
       nb_counter=0        

• Part 4 : Appending 'dash' or 'dot' to a list for long/short breath

When long or short breath is detected, a dash or dot is appended to a list named 'msg_array'.If it is the first time appending then, it is done directly. For all other attempts the corresponding '.' or '-' is concatenated for seamless match detection of morse code, which we will be seeing in the comings steps. A counter named 'list_counter' is used to keep track of the appending attempts.

    if(gen_counter==15):    
       if(b_counter>0 and nb_counter>0): 
           if(b_counter>nb_counter):     
               print('Long Breath') 
               lb_counter+=1 
               if(list_counter==0):    
#appending '-' or 'dash' into msg_array when long breath is detected 
                   msg_array.append('-') 
                   msg_counter+=1 
               else: 
                   msg_array = [x + dash for x in msg_array]    #concatination of '-' 
               list_counter+=1     
           else:    
  
               print('Short Breath') 
               sb_counter+=1 
               if(list_counter==0):    
#appending '.' or 'dot' into msg_array when long breath is detected 
                   msg_array.append('.') 
                   msg_counter+=1 
               else: 
                   msg_array = [x + dot for x in msg_array]    #concatination of '.' 
               list_counter+=1 
       if(b_counter==0):  #keeping track if user stops breathing             
           bn_counter+=1         
       gen_counter=0 
       b_counter=0 
       nb_counter=0 

• Part 5 : Conversion from Morse code to English alphabets

To trigger the conversion process the user has to stop breathing for 3 attempts or the Morse code combinations should add up to 5 (Morse can have only maximum 5 combinations of '.' and '-' ).

When this is detected the conversion takes place. Below I've shown one example of how I'm converting from Morse code '.-' to alphabet 'a':

   if(lb_counter+sb_counter>=5 or bn_counter>=2):    
#when the morse inputs add up to 5 or the user stops breathing 
#for 3 attempts then morse code checking and conversion takes place 
       print(msg_array) 
       if (findexact(msg_array,m_a)==1):    
#checking for morse combination match and 
#if true, text to speech speaks the correponding english alphabet 
           print('A') 
       
       else:    #if no match found     
           print('Not valid morse combination') 
       clear_ar() 
       pt_counter=0 
       msg_counter=0 
       lb_counter=0 
       sb_counter=0 
       bn_counter=0 
       list_counter=0  

The findexact() function checks for an exact match between the msg_array user input and 'm_a' string where m_a = ['.-'] which is Morse for 'a'.

def findexact(msg_array,morse):    
#a function that checks if content of msg_array and morse are an exact match 
   i=0 
   while i < len(msg_array): 
       if any(item == msg_array[i] for item in morse): 
           return 1 
       i+=1 

Additionally, once the conversion is done the list is cleaned using the clear_ar() function:

 def clear_ar(): 
   del msg_array[:] 

• Part 6 : Appending Letter to 'wrd_array' and using Text To Speech

Once the conversion is done, then E-speak text to speech is used to speak out the alphabet, and then the 'English Alphabet' is inserted to another list namely 'wrd_array'.

       if (findexact(msg_array,m_a)==1):    
           print('A') 
           os.system("espeak 'A'") 
           insert_wrd(wrd_array,l_a) 

The insert_wrd() function does a similar job of the appending and concatenation part we've seen in Part 4. But, in this case the alphabet is appended.

def insert_wrd(wrd_array,letter):    #a function that concatinates word input 
   food = ['FD'] 
   hay = ['HAY'] 
   iaf = ['IAF'] 
   wt = ['WT'] 
   global wrd_counter 
   if(wrd_counter==0): 
       wrd_array.append(letter) 
   else: 
       wrd_array = [x + letter for x in wrd_array] 
   wrd_counter+=1  

Speaking large commands can often be difficult to breathe out in Morse for the user, so I took the most commonly used sentences like 'How are you?' , 'I am fine', 'I want food' , 'I want water' and made short Morse commands for these big sentences.For eg : 'FD' stands for 'I want food' as seen in the video.

Once the alphabets are appended, there is a frequent check done on the wrd_array to see if it matches with any of the commands that we've created.

    
def findexactword(wrd_array,word):    
#a function that checks if content of wrd_array and word are an exact match 
   j=0 
   #print('Inside Find Exact Word') 
   while j < len(wrd_array): 
       if any(items == wrd_array[j] for items in word): 
           return 1 
       j+=1 
def insert_wrd(wrd_array,letter):     
   food = ['FD'] 
   hay = ['HAY'] 
   iaf = ['IAF'] 
   wt = ['WT'] 
   global wrd_counter 
   if(wrd_counter==0): 
       wrd_array.append(letter) 
   else: 
       wrd_array = [x + letter for x in wrd_array] 
   wrd_counter+=1 
   if (findexactword(wrd_array,food)==1): 
       print('I want food') 
       os.system("espeak 'I want food'") 
       wrd_counter=0 
       clear_wrdar() 

That pretty much sums up the main parts of them program, sorry about the variable names, I'm not that good with naming. I hope you've been able to get a good understanding of the main parts, if not let me know your doubts in the comment section below.

The entire source code is available below for your reference.

Conclusions

WalaBreathe proves to be an efficient assistive device as:

• It is wireless

• Uses RF reluctance data to classify long and short breaths.

• Low cost and compact in form factor compared to existing alternatives.

• Faster than existing assistive devices

Working on this project gave me new insights into programming RF Imaging sensors and also allowed me to explore the various parts of Python programming.

I would like to thank Noam Hoze from the Walabot team who approved my idea and enabled me to work on the walabot. Special thanks to Sachar Or from Vayyar who introduced me to Pyplot!

I would like to also thank Adam from Hackster who shipped the Walabot to me, Thank you Adam!! This is one among the best hardware that I've worked with!

Thank you all for checking out the project, Hope you enjoyed reading through/building this project.

Have a nice day!