Edge Impulse Pill Identifier

Have you ever taken a pill out of the container it came in and immediately forgotten the name of the pill? I have, especially when I organize the pills I need to take each day into a daily container.

I take a lot of vitamins, so I don't worry too much about when I take my pills or in what order. But what if you're one of the millions of people who takes multiple pills each day for things like diabetes, cancer, high blood pressure, and other medical issues? It's probably critical that you organize your pills according to how your doctor tells you to take them. Mixing up pills, taking them at the wrong time (time of day/night, with/without food, with/without other medications), and generally not knowing what pills you have once they are removed from the pill bottles are serious issues.

My mother takes about 20 pills a day. I am worried that she won't always know which pills she has to take, so I looked at pill identification apps and websites. Every one of them requires you to be connected to the internet.

And that's a problem for many people. It may come as a suprise to you, but there are millions of people in the U.S. who do not have any type of internet connection at home. The largest group is older adults. (See https://www.pewresearch.org/internet/fact-sheet/internet-broadband/ for more information.) And that's the group that is most likely to take medication.

So I set out to build a pill identifyer that can run on a phone when it is not connected to the internet.

Project planning:

1. I took as many bootcamps as I could about Python and machine learning. For this project, the best course I took was Computer Vision with Embedded Machine Learning, a free course on Coursera taught by Shawn Hymel (his Twitter handle is @ShawnHymel). Included in this course were lectures on how to code in Python as well as how to use Edge Impulse to create an image recognition neural network model.

2. I also read as much as I could and watched videos/webinars on edge technology. If you haven't heard of this, you need to learn about it immediately. You'll fall down the rabbithole when you search because you'll come across terms like edge technologies, tinyML, embedded machine learning, and more.

Devices that run on the edge can include your phone, wearable devices like smart watches, sensors, autonomous vehicles, and other IoT devices. Simply stated, edge computing can be done on the device itself; you don't need to send information to the cloud.

That's an enormous benefit because edge computing reduces latency (delays) when it is processing information. For my purposes, I'm most interested in the ability of edge devices to not be connected to the Internet because of the problems I mentioned above. Millions of people do not have reliable internet, or high speed internet, or any internet at home. Yet there are ways to build solutions based upon running computations on edge devices.

Project building:

3. To create my dataset, I borrowed 6 pills from my mother. I chose to only photograph 6 different pills of brandname drugs for this project because they were ones I could photograph myself; I had access to them.

Trying to find multiple copyright-free photos of the same pills to use in a machine learning model was impossible. I also had spent a day looking online on webmd at various pills. Silly me, I didn't know just how many different shapes, sizes, colors, and identifying marks each pill had.

So I decided to photograph the pills myself.

When thinking about my dataset, I thought about how a person would hold their phone when using it to classify their pills. They would probably place the pill on a plate, or on a countertop. So the backgrounds would be varied. They may or may not have the pill centered in the frame. Focusing on the pill would not be a given. They may hold their phone at different distances from the pills. There will certainly be a difference in the lighting captured and the cast shadows produced from the light source.

I decided I needed a lot of photos. There were several ways I obtained photos of the pills...

4. Photographing pills using my phone: I used my phone to photograph each pill many, many, many times. I placed the pills separately on different countertops, at different angles, at different sizes in the frame, at different locations in the frame. I then brought these images into Photoshop to see how they looked. I saved them as png files and uploaded these images to Edge Impulse.

5. Data augmentation: I wrote Python code in Jupyter Notebooks to perform data augmentation on the images I took of the pills.. I rotated images, moved them around in the frame, changed the brightness and darkness, blurred some, and added noise to others. These files were also saved as png files and also uploaded to Edge Impulse.

6. Directly uploading photos to Edge Impulse: Edge Impulse allows you to connect your phone to your project, and then use your phone to upload photos to their website. These photos were ultimately saved as jpgs.

In the end, I wound up with 1189 photos for my training and validation dataset, and 305 photos in my testing dataset.

7. I then built my model in Edge Impulse. After a lot of work on my training, validation, and testing various models and looking at accuracy rates, I landed on the following for my neural network model:

This model resulted in an accuracy rating of 87.8% and a loss of 0.47. The confusion matrix was ok and I knew I may have problems with a few of the pills based upon what it showed me.

I chose to create a custom deep neural network instead of using a pre-trained transfer learning neural network because the accuracy was much better on my custom network. And my model performed better when the features of the images were kept as RGB instead of converting them to grayscale.

Project Testing:

8. Testing on my testing dataset: Of the many bootcamps I've taken, I've heard people say they've tested their machine learning model by taking a separate part of a dataset and removing it completely so the model does not use the data to train on. Then, this second dataset is used after a model is trained to "test" the model.

Edge Impulse lets you "test" your model in a way similar to how you'd test it if you were writing straight Python code. My testing resulted in an accuracy reading of 88%, which was pretty good. But the real test was deploying this model to my phone and using my phone to classify the pills.

9. Real world testing... I placed the pills on a countertop and used Edge Impulse to connect my phone to it. Yes, I needed the Internet here because Edge Impulse was hosting my files. They packaged up all of the files and built an app the works in a phone's browser. Once the files are deployed, though, you can disconnect from the Internet and the classification still works! Your phone is being used as an edge device running a machine learning model without having to send information to the cloud (and back) in order for it to work.

There is definitely room for me to try to improve my model's accuracy, expand to other edge devices, expand the number of pills that can be classified, etc. But overall, I'm happy with the results of the project and learned a lot from creating it.