AdaptoPharm: An Adaptive Pharmacy Inventory System

Introduction

The local equine hospital my wife works for is having difficulty managing inventory for their pharmacy which drastically lessens their productivity due to having a reduced staff and an increased workload since the onset of COVID-19.

This project aims to demonstrate the practicality of Adaptive IOT by creating an inventory management system that incorporates machine vision to predict which medicine is withdrawn from the pharmacy, determine when medicine needs to be ordered, and then generate and send a report to the employee for review at the end of each day.

Please note that this is an engineering build and is not intended to be fielded until it has gone through more rigorous development and testing. The main goal of this guide is to show how quickly one with limited knowledge on AI and the Xilinx platform can put together an Adaptive IOT solution.

Planning

Initially the plan was to use a bar code scanner to record the medicine being withdrawn and deposited into the pharmacy as they are commonly found within most businesses. However there were two issues with this:

Barcode scanners are expensive.

Not all medicine bottles have barcodes.

Instead of barcodes they have what is called a National Drug Code (NDC). An example can be seen below.

After eliminating the option of a barcode scanner the second concept that came to mind was using RFID.

An awesome example can be seen here: Smart-Medicine-Control - Hackster.io

However after some thoughtful consideration, I eliminated this concept as well. This was because I believed it would be too much of a hassle for the person in charge of the pharmacy to keep inventory of the medicine and the supply of RFID stickers that would have to be placed on each bottle. Along with having to keep track of which RFID represents their respective medicine.

The chosen solution was to defer the proposed AI powered inventory optimization and use machine vision to process images of the medicine with the use of the Ultra96V2's DPU.

Step One: Setting Up the Ultra96v2

First we need to install PYNQ 2.5 and Vitis AI 1.1 onto our Ultra96v2 board. This can be done by following Wadulisi's excellent guide found here: Easy AI with Python and PYNQ - Hackster.io

Be sure to run the various examples within that guide to ensure that everything is working properly!

Step Two: Gather the Data for Training

After having setup the board, it's time to collect various images of our randomly generated medicine names. I used Medicine name generator (fantasynamegenerators.com) to create five names and then with the help of Microsoft PowerPoint I created generic labels as shown below.

Next we tape the labels over the empty medicine bottles and begin collecting images using the Python script gather_data.py on our PC. Ensure that you have created two folder directories in /tmp, one for training and the second for testing. I saved 100 images of each medicine in /tmp/train/medicinename and ten images of each medicine in /tmp/test/medicinename.

Note: You will probably need to edit line 11 /dev/video# the # represents which port your webcam is on.

This script will prompt the user in the command line three questions. Their expected input is given below in an example:

• Where am I saving images? tmp/train/abibicin
• How many images are needed? 100
• What is the medicine name? abibicin

Once the inputs are given, the script will begin taking pictures and you will be prompted to press enter to continue. This was put in the script to make it easier to rotate the bottle in-between the capturing of images.

Repeat this step for each type of medicine in our sample set. An example of some of my images captured are shown below. Each row is a different type of medicine starting from the top Abibicin, Secrefoxin, Abirafenac Edrozine, Gluataine, and Tracstral.

Step Three: Training for DPU Compilation

In this step, initial training was done by adapting this guide 10 Minutes to Building a CNN Binary Image Classifier in TensorFlow | by Binh Phan | Towards Data Science for two of our medicines (Abibicin & Secrefoxin).

Edit for where your images are located on your PC and run train.py to create a tensorflow model. Our initial Receiver Operating Characteristic (ROC) is shown below.

We also evaluate the effectiveness of our model within the same script by loading ten different pictures of abibicin and then ten different pictures of secrefoxin into the model. Our results are shown in the terminal with abibicin only being detected as secrefoxin twice.

abibicin4.jpg is abibicin
[1.]
abibicin9.jpg is abibicin
[1.]
abibicin0.jpg is abibicin
[1.]
abibicin7.jpg is abibicin
[1.]
abibicin3.jpg is abibicin
[1.]
abibicin5.jpg is abibicin
[0.]
abibicin2.jpg is secrefoxin
[1.]
abibicin8.jpg is abibicin
[1.]
abibicin6.jpg is abibicin
[0.]
abibicin1.jpg is secrefoxin
[1.]
secrefoxin5.jpg is secrefoxin
[1.]
secrefoxin0.jpg is secrefoxin
[1.]
secrefoxin4.jpg is secrefoxin
[1.]
secrefoxin1.jpg is secrefoxin
[1.]
secrefoxin3.jpg is secrefoxin
[1.]
secrefoxin6.jpg is secrefoxin
[1.]
secrefoxin9.jpg is secrefoxin
[1.]
secrefoxin2.jpg is secrefoxin
[1.]
secrefoxin7.jpg is secrefoxin
[1.]

After training the model we need to follow this guide DPU-PYNQ/host at master · Xilinx/DPU-PYNQ · GitHub to build the model for the DPU.

Step Four: Creating the GUI and Basic Inventory Report

Using Tkinter we can create a basic GUI for the user. The resulting GUI is capable of taking images, recording the medicine name, and its location. Eventually the Detect pushbutton will populate the medicine name textbox with the predicted name. This textbox can still be edited by the user however in case the model were to predict the wrong medicine.

By hitting the Send Report button an email will be sent to the specified emails that are saved in the mycontacts.txt. The emailing capability was adapted from Arjun's great guide: Send Emails Using Python (freecodecamp.org)

The resulting report is shown below!

Future Plans

• Get Detect Pushbutton working
• Update Tensorflow model from a Binary CNN to a Categorical CNN
• Update Inventory Management Algorithm to use AI
• Add more sensors! (e.g. Temperature & Humidity monitor, Motion Sensor for when controlled substance safe is opened)
• Upgrading the system to be more compatible with communicating with cloud based Veterinary or Human Medicine Practice Management Software