According to the World health organisation (WHO) nearly 50 million people worldwide suffer from dementia, with the most common form of the disease being Alzheimer's which accounts to 60-80% of dementia cases.
Having this leaves the person unable to remember important day to day tasks that needs to be completed. One of the common task that people with dementia face is forgetting to take the required medications and being unaware of this.
Automatic pill dispensers have been of great help, but with advances in speech recognition and machine learning there are possibilities to build innovative pill dispensers with which a user can interact and keep track of various statistics corresponding to daily pill/medicine intake.
What is Curismo?Curismo is a smart pill box that is built on top of the Walabot Pro and Amazon Alexa. Curismo is a portmanteau of Curis (which means healthcare in latin) and monitoring.
Curismo detects when a user places their hand on top of the pill box and based on the location of the hand above the pill box (X, Y, Z) and the height between the hand and sensor (<20cm) the pill which is taken is determined by tracking the distance of the hand from the sensor placed under the pill box. Once, the amount of pills taken are logged using the Walabot sensor data, the user can interact with a custom Curismo Alexa skill to keep track of how many pills they have taken, how many pills are remaining to be taken and the time at which the pills were last taken. The user can interact with the skill and get insights to make the right decision by enabling a user to store their prescription and reach their daily pill intake goals with the help of the Curismo Alexa skill which tells how many pills to take and when to take them next.
Hardware Required and Setup:1) Walabot Pro (Link)
The instructions to setup Walabot on your PC is available on the Walabot page under the Quick Start Guide: https://walabot.com/getting-started .
2) Amazon Alexa enabled Hardware/Simulators:
If you do not have an amazon echo the alternatives to test the skills are Echo Dot/Echoism/Alexa Developer Kit Simulator.
I personally did all the testing on the amazon Developer console which comes in really handy :)
3) A box to place with enough area to place the Walabot inside and cardboard pieces that can be used divide the placement of pills into different sections.
Software Setup and Requirements:The software development process is broken down in three parts, the first part covers how to build the program that detects from which direction the hand is approaching and which pill has been taken.The Walabot python program also stores the information about pill being picked and the time at which it was picked and stores it into two separate text files for later use by the Amazon Alexa Python skill.
Next, we'll be discussing how to build a custom amazon skill for Curismo using Flask-Ask and Python. And finally, we'll be looking into how to design the front-end Voice User Interface (VUI) of the Curismo Alexa Skill.
Part 1 : Working with Walabot1. Walabot SDK Introduction:
The Walabot sensor gives a programmer the ability to program/work with the raw signals in various ways. It is surely going to be a super-fun experience programming the Walabot sensor and I'm sure it can help shape up your ideas.
The Walabot SDK is available for both Windows and Linux, I worked using the Linux SDK on Lubuntu.
2. Setting up Walabot with Python:
For this project I mainly used Python, for setting up Walabot with python. To install python you can refer this link: https://www.python.org/downloads/
Once that's done, you can download the Walabot SDK for your OS. In my case I downloaded the .deb file and installed it via the terminal. Detailed information about the Walabot Python API is available at: https://walabot.com/api/_pythonapi.html
When starting the project I was looking for a way to track a users hand, a way to visualise how Walabot tracks an object. To get a visual understanding I mainly referred the Walabot Sensor Targets repository created by gal-vayyar on github.
It can be found here https://github.com/Walabot-Projects/Walabot-SensorTargets
What followed next was building the Alexa skill that works together with the Walabot to help the user keep track of their pill intake.
3. Writing the logic for Curismo Python Program:
Curismo was built mainly on top of the Walabot sensor targets project. Modifications where made so as to detect from which location the hand is approaching and the height at which the target is located at in real time.
To understand how to classify between hand approaching from left, centre or right an observation was made after observing the Walabot target tracking visualiser.
By looking at the target visualiser, it is clear that when the arm approaches from the left, the Y-axis magnitude tends to go towards -ve values, when at the centre the y-axis value is between -5 to 5 and towards the right the Y-axis values are +ve and usually greater than 10 for object at far right.
Similarly, the z-axis gives the magnitude in centimetre as to the distance of the target from the sensor. In order to confirm that the user has taken a pill the z-axis activation threshold was set to be around 20cm, so that only when the user approaches to a close proximity does the counter corresponding to a particular pill undergoes an increment by 1.
Now, that we are able to detect if the hand approached from left, right or centre we can move onto making sure erroneous outputs are avoided. In the program we track the Z-axis (height) distance between the hand and sensor increasing as user takes the hand back after picking up a pill. This is used to confirm that the user has already picked up the pill/medicine at least once.
This was implemented in the program as shown below.
def update(self, targets):
.
.
.
if (targets[i].zPosCm<20): #if hand is under 20cm range
if(targets[i].yPosCm>10): #if hand is inside right section
r_counter+=1
if(r_counter==1):
rm_counter+=1 #right side pill value incremented
print('Right Enterances : ',rm_counter)
elif(targets[i].yPosCm<-10):
l_counter+=1
#print('User Approaching from Left')
if(l_counter==1):
lm_counter+=1
print('Left Enterances : ',lm_counter)
elif(targets[i].yPosCm>-5 and targets[i].yPosCm<5):
m_counter+=1
#print('User Approaching from centre')
if(m_counter==1):
mm_counter+=1
print('Middle Enterances : ',mm_counter)
elif (targets[i].zPosCm>30):
open('data.txt', 'a').close()
open('datatime.txt', 'a').close()
r_counter = 0
l_counter = 0
m_counter = 0
txt = "#{}: x: {:3.0f} y: {:3.0f} z: {:3.0f}".format(
i+1,
targets[i].xPosCm,
targets[i].yPosCm,
targets[i].zPosCm)
self.targetLabels[i].config(text=txt)
else:
open('data.txt', 'a').close()
open('datatime.txt', 'a').close()
self.targetLabels[i].config(text="#{}:".format(i+1))
4. How is the pill Intake information written into a file?
Once the logic for tracking and counting the number of pills that have been taken from the box, we need to next log the details/store the pill box information so that the Alexa skill can access the stored information and use it to provide useful analytics regarding a users pill intake.
#if
f.write("rm : "+str(rm_counter)+' '+'\n') #info log
f.close()
#storing current time to another file
t = datetime.now()
f = open("datatime.txt", "a")
f.write("rm : "+str(t.hour)+':'+str(t.minute)+'\n')
f.close()
For this, I have created two text files, one named 'data.txt' which stores the number of a particular pill taken from the box, the value is labelled with 'lm' , 'rm' and 'mm' keys which are used to uniquely identify that the pills where taken from left section, right or centre section of the pill box.
Next, another file named 'datatime.txt' stores the exact time at which a particular pill was taken from the box. This data can be used in the alexa skill to let the user know when he had last taken a particular pill.
This is written in the program as follows:
if(targets[i].yPosCm>10): #if hand is inside right section
r_counter+=1
if(r_counter==1):
rm_counter+=1 #right side pill value incremented
f = open("data.txt", "a")
f.write("rm : "+str(rm_counter)+' '+'\n') #info log
f.close()
#storing current time to another file
t = datetime.now()
f = open("datatime.txt", "a")
f.write("rm : "+str(t.hour)+':'+str(t.minute)+'\n')
f.close()
print('Right Enterances : ',rm_counter)
elif(targets[i].yPosCm<-10):
l_counter+=1
#print('User Approaching from Left')
if(l_counter==1):
lm_counter+=1
f = open("data.txt", "a")
f.write("lm : "+str(lm_counter)+'\n')
f.close()
t = datetime.now()
f = open("datatime.txt", "a")
f.write("lm : "+str(t.hour)+':'+str(t.minute)+'\n')
f.close()
print('Left Enterances : ',lm_counter)
elif(targets[i].yPosCm>-5 and targets[i].yPosCm<5):
m_counter+=1
#print('User Approaching from centre')
if(m_counter==1):
mm_counter+=1
f = open("data.txt", "a")
f.write("mm : "+str(mm_counter)+'\n')
f.close()
t = datetime.now()
f = open("datatime.txt", "a")
f.write("mm : "+str(t.hour)+':'+str(t.minute)+'\n')
f.close()
print('Middle Enterances : ',mm_counter)
elif (targets[i].zPosCm>30):
open('data.txt', 'a').close()
open('datatime.txt', 'a').close()
r_counter = 0
l_counter = 0
m_counter = 0
txt = "#{}: x: {:3.0f} y: {:3.0f} z: {:3.0f}".format(
i+1,
targets[i].xPosCm,
targets[i].yPosCm,
targets[i].zPosCm)
self.targetLabels[i].config(text=txt)
else:
open('data.txt', 'a').close()
open('datatime.txt', 'a').close()
self.targetLabels[i].config(text="#{}:".format(i+1))
Finally, the data files are read by the python Alexa skill built using flask-ask.
Part 2: Amazon Alexa Skill Development1. Setting up a developer account and understanding Alexa skill development.
The first step is to head over to https://developer.amazon.com/ and create an amazon developer account. This enables you to gain access to develop Amazon Skills. When getting started with Alexa I first refereed the free Codeacademy course on developing Alexa skills. It can be found here: https://www.codecademy.com/learn/learn-alexa
Or if you would like other options you can check out: https://developer.amazon.com/alexa-skills-kit/alexa-skills-developer-training#Beginner
After working on some sample projects, I releasing that the best way to improve my workflow would be to start writing Alexa skills using Python so that I could directly connect the Walabot python code and amazon Alexa skills code.
The perfect framework for this is Flask-Ask by John Wheeler, which is a python library that enables you to develop amazon Alexa skills.
2. Building a custom Alexa Skill using Flask-Ask
Flask Ask is a great resource to work with if you are a python developer. To get started with flask ask you can refer: https://developer.amazon.com/post/Tx14R0IYYGH3SKT/Flask-Ask-A-New-Python-Framework-for-Rapid-Alexa-Skills-Kit-Development
John Wheeler has provided some great tutorials on his flask ask page as well, https://alexatutorial.com/flask-ask/.
3. Writing the Curismo Alexa skill using Python
The main features that I've included in the Curismo skill are:
- Ask Curismo How many pills are already taken
- How many pills are left to be taken
- At what time did I have the pills at last (Keeps track of time at which user had pill previously)
- What is my prescription (medicine with food or without food)
First make sure all the python files are in the same folder. Create two empty text files named 'data.txt' and 'datatime.txt' or any other name you prefer. Next, create a file named templates.yaml and include the follow code for welcome message.
welcome: Welcome to curismo! I am your smart pill box assistant.
Next, create a new python file which will be the source of our custom amazon skill.
At first we need to import some necessary libraries:
import logging
from flask import Flask, render_template
from flask_ask import Ask, statement, question, session
from dateutil.parser import parse
#for parsing time of pill intake recorded using walabot
import re
from datetime import datetime
Next, we create a set of variables onto which the pill information like (count, time) are stored into by reading the data.txt and datatime.txt files with information logged using Walabot sensor data.
#stores count of pill 1, 2 and 3 respectively
p1 = 0
p2 = 0
p3 = 0
#used while getting time and count substrings from files
v = 0
i =0
match = 0
#used to store time logs read from file
p1time = 0
p2time = 0
p3time = 0
#used to set prescription of the user for a given pill
prescp_p1 = 3
prescp_p2 = 3
prescp_p3 = 3
The values of these variables are assigned based on reading the data and datatime text files.
4. How is the pill Intake information written into a file?
Two functions readnupdate() and readpilltiming() are used to reading pill intake no and timing respectively. readnupdate() function reads data.txt file and stores the count of pill 1, pill 2 and pill 3 taken based on information logged by the Walabot sensor.
def readnupdate():
global p1,p2,p3,v,i
with open('data.txt', 'r') as f:
#file with pill intake number opened in read mode
while True:
if(i==3): #as we are having only three pills
i = 0
if(i<3):
v = f.readline()
if not v: break
if("lm : " in v):
#unique label "lm" used to filter left section pill intake
[int(s) for s in v.split() if s.isdigit()]
p1 = int(s)
elif("mm : " in v):
[int(s) for s in v.split() if s.isdigit()]
p2 = int(s)
elif("rm : " in v):
[int(s) for s in v.split() if s.isdigit()]
p3 = int(s)
f.close()
readpilltiming() stores time as string to variables p1time, p2time and p3time based on time information corresponding to time at which the user took a pill. The timestamp is written to the file in the Curismo Walabot program. Here, we are reading to the time data logged by the Walabot program so as to use it in the Alexa skill for letting the user know when he had last had a particular pill.
def readpilltiming():
global p1,p2,p3,v,i,match,p1time,p2time,p3time
with open('datatime.txt', 'r') as f:
while True:
if(i==3):
i = 0
if(i<3):
v = f.readline()
if not v: break
if("lm : " in v):
match = parse(v, fuzzy=True)
p1time = datetime.strptime(str(match.hour)+':'+str(match.minute), '%H:%M').strftime("%I:%M %p")
elif("mm : " in v):
match = parse(v, fuzzy=True)
p2time = datetime.strptime(str(match.hour)+':'+str(match.minute), '%H:%M').strftime("%I:%M %p")
elif("rm : " in v):
match = parse(v, fuzzy=True)
p3time = datetime.strptime(str(match.hour)+':'+str(match.minute), '%H:%M').strftime("%I:%M %p")
f.close()
Next, we create a function that defines the event that occurs during launch.
app = Flask(__name__)
ask = Ask(app, "/")
logging.getLogger("flask_ask").setLevel(logging.DEBUG)
@ask.launch
def starting():
readnupdate()
welcome = render_template('welcome')
return statement(welcome)
Next, I created an intent that is activated when the user asks "how many pills did I have today".
@ask.intent("AllPillIntent")
def allpill():
readnupdate()
global p1,p2,p3
return statement('as of now, you have had {} pill 1, {} pill 2 and {} pill 3'.format(p1,p2,p3))
The next Intent, reads out the daily prescription based on information provided by the user and also informs the number of pills that have been taken at the moment.
@ask.intent("DailyPrescpIntent")
def dailypres():
readnupdate()
return statement('As per your prescription, you need to have {} pill 1 with food, {} pill 2 and {} pill 3 without daily. As of now you have had {} pill 1, {} pill 2 and {} pill 3'.format(prescp_p1,prescp_p2,prescp_p3,p1,p2,p3))
The next intent is used to inform the user about any pill that has not be taken even once.
@ask.intent("PillNotTakenIntent")
def pillnothad():
readnupdate()
global p1,p2,p3
if(p1==0 and p2==0 and p3==0):
return statement('you have not had any pills today')
elif(p1==0 and p2==0 and p3!=0):
return statement('you have not had pill 1 and pill 2')
elif(p1==0 and p2!=0 and p3==0):
return statement('you have not had pill 1 and pill 3')
elif(p1!=0 and p2==0 and p3==0):
return statement('you have not had pill 2 and pill 3')
elif(p1==0 and p2!=0 and p3!=0):
return statement('you have not had pill 1 today')
elif(p1!=0 and p2==0 and p3!=0):
return statement('you have not had pill 2 today')
elif(p1!=0 and p2!=0 and p3==0):
return statement('you have not had pill 3 today')
else:
return statement('you have atleast had one of each pill today')
The next Intent is activated when the user asks "When did I last have a pill". It collects time information logged by the Walabot program and then informs user about the timings at which the particular pills where taken at.
@ask.intent("PillTimeIntent")
def pilltiming():
readpilltiming()
global p1time,p2time,p3time
return statement('you had pill 1 at {} , pill 2 at {} and pill 3 at {}'.format(p1time,p2time,p3time))
Finally, the last intent informs the user about how many pills are left to be taken by calculating the different between the prescribed number of pills and the amount of pills taken as of that moment.
@ask.intent("PillRemainingIntent")
def pillremaining():
readnupdate()
global prescp_p1,prescp_p2,prescp_p3
r1 = prescp_p1 - p1
r2 = prescp_p1 - p2
r3 = prescp_p1 - p3
return statement('you have {} pill 1, {} pill 2 and {} pill 3 left to be taken today'.format(r1,r2,r3))
Now, that we've written the Curismo Walabot program and Alexa skill, we can move onto linking the python program with Alexa using the Alexa Skill Builder.
Part 3: Setting up Alexa Skill on Skill BuilderFirst log into your amazon developer account and the head over to this page: https://developer.amazon.com/alexa/console/ask
From here, click on create a new skill. If you're using the new Skill Builder UI Console then you can follow right away, it is quite the same in the old UI as well. Additionally, I'll be including the Intent Schema for the old UI as well.
Make sure that the invocation name is Curismo.
Next, go to the Intents tab, there you'll need to create 5 new intents. Which are as follows:
For those using the old UI, the Intent Schema is as follows:
{
"intents": [
{
"intent": "AllPillIntent"
},
{
"intent": "AMAZON.CancelIntent"
},
{
"intent": "AMAZON.HelpIntent"
},
{
"intent": "AMAZON.StopIntent"
},
{
"intent": "DailyPrescpIntent"
},
{
"intent": "PillRemainingIntent"
},
{
"intent": "PillNotTakenIntent"
},
{
"intent": "PillTimeIntent"
}
]
}
The sample utterances are:
AllPillIntent how many medicines did i have today
AllPillIntent how many tablets did i take today
AllPillIntent how many pills did i take today
AllPillIntent how many medicines did i take today
AllPillIntent how many pills did i have today
AllPillIntent how many tablets did i have today
DailyPrescpIntent how many pills do i have to take per day
DailyPrescpIntent what is my daily prescription
PillRemainingIntent how many more pills should i take today
PillRemainingIntent how many pills are remaining to be taken today
PillRemainingIntent how many pills do i have to take today
PillNotTakenIntent what pills did i forget to take today
PillNotTakenIntent what medicines did i not take today
PillNotTakenIntent what pills did i not take today
PillTimeIntent at what time did i have my pills
PillTimeIntent when did i last take my pills
PillTimeIntent at what time did i have my medicines
PillTimeIntent when did i take my medicines at last
Once you've created the Intents make sure you save your model and click on Build Model. Next head over to the configuration tab, in the Endpoints section under the Service Endpoint make sure you select "HTTPS" instead of Amazon ARN.
Then next, you'll need to open up terminal and type in (for linux, varies in windows and mac):
./ngrok http 5000
Then copy the final Forwarding URL and paste it in the Default URL field under service endpoint. This ensures that the Curismo Alexa Python Skill is linked with Alexa online via http port 5000.
Click next, and finally under global fields make sure you check:
My development endpoint is a sub-domain of a domain that has a wildcard certificate from a certificate authority
Congrats, now you're all set to test the Curismo Alexa Skill.
For ensuring that everything works well, follow the sequence given below.
- Connect Walabot to PC, place Walabot inside the box and pills on top of the box.
- Open curismowalabot.py file and run it in IDLE or any other IDE.
- Open terminal and run
./ngrok http 5000
- Open another terminal tab and run python curismoalexa.py
- Finally open up the online Alexa Skills simulator and say Open Curismo.
The commands that you can try out are:
open curismo
alexa ask curismo how many pills did i have today
alexa ask curismo what pills have i not had today
alexa ask curismo when did i last have my pills
alexa ask curismo at what time did i last have my pills
alexa ask curismo what is my daily prescription
alexa ask curismo how many pills do i have to take today
Conclusion:Curismo proves to be an efficient pill/medicine intake monitoring tools. It is the first of it's kind Pill box that allows patients with dimentia to interact with the device using Alexa services and get insights into:
- How many pills the user has already taken
- How many pills are remaining to be taken
- At what time did the user last take the pills
- What is their daily prescription and how close they are to reaching their daily medication intake goals.
I'm optimistic that such systems where user-computer interaction plays a crucial role can shape how patients with dementia attend to their medication on a daily basis.
Hope you enjoyed the build, you can ask your queries in the comments below or through DM.
Comments