OneMed

Why? The Background Story

Being a part of a family of doctors, I've seen medical conditions first-hand. I've stayed up nights and heard my parents talk about a certain patient and the procedure they need to follow while diagnosing/operating the next day. It's amazing how much more efficiency we can bring in medical fields with the help of machine learning and deep learning technologies. Healthcare is probably one of the biggest industries in the world and still falls short of quality supply of medical professionals. Looking forward, deep learning can have a huge impact on assistive medical usecases.

How? OneMed

I've been researching about the intersection of medical science and deep learning for quite some time now. This time, I decided to build tools that can be used in real life. OneMed is a group of medical tools based on the web which allow doctors to perform various types of analysis only through an image. It uses core computer vision technology for classification of images and speeds up processing using OpenVino.

I needed to design something that even a beginner can use in no time. OneMed consists of three main sections as of now: EyeMed, COVID-Med and Skin Med.

EyeMed performs functions around the eye like cataract detection, diabetic retinopathy detection and redness levels (uveitis) detection.

COVID Med occurred to me because as an Indian, I've seen the shortage of medical expertise in the rural areas. COVID Med analyses CT scans of lungs for detection of COVID-19.

Skin Med detects malignant and benign type of skin cancer.

A lot of this medical technology is very intensive and requires co-ordination of multiple things. Understanding the medical images, converting them into machine data and then carrying out multiple stages of processing on it. OpenVino has been an incredible framework to put this together.

What? The Tech

The models are trained on core computer vision and image analysis technology of Keras and tensorflow engine. The models behind these functions were trained on Tensorflow + Keras and then converted to Openvino IR format. There're simple deep learning processes that back the core models

The front-end of the web app is built on Streamlit which is an open source data framework which flexibly allows developers to push out their machine learning/deep learning code to the web.

Video Demo

• Fast and accurate analysis.
• Making healthcare more accessible to the world's growing population.
• Assistive analysis for doctors.

The Making

Step 1 : Making, tuning and converting the models.

The backend runs on a classification algorithm. It's trained to differentiate between 2-3 classes of medical conditions. The classification has an accuracy of above 85% leaving aside exceptional cases.

Models for this project were trained using Teachable Machine which is a low-code tensorflow/keras-based model builder. All models were custom trained for the use-cases. The models were trained on about 100-150 images for each class of medical problems. The models have been fine-tuned and trained on different variations of epochs and batches to achieve maximum accuracy (For eg: the COVID scan analysis model has been trained on 3 epoch variations and the most accurate one was 38 epochs, which is used in the project) Then these were downloaded the models locally for further development. The models were downloaded in Keras h5 format.

Why OpenVino ?

I've used a custom shell script to convert the keras.h5 models to Openvino IR format. The script was originally written by Ojas Sawant who works at Intel.

Most doctors who I've interacted with in the past have shared a common complaint with regards to speed. They don't have enough time to wait for deep learning models to show their inferences. The medical professionals need to focus more on their patients and the software tools need to be a part of their flow. That's where the agility and speed of OpenVino comes in. Faster inferencing is the biggest benefit. Saving those few seconds while running the models is a huge benefit when compounded to 1000s of patients. I have got a bunch of different models at play here and so I don't have exact improvement metrics, but I'm pretty sure it is atleast 10x faster than Keras alone.

Code and models available on My GitHub Repository.

Step 2: Front-end with streamlit.

Streamlit is a powerful front-end framework for operating ML/data apps. The entire front-end of this project is made using Streamlit. As the project involved a lot of complex co-ordination between models, front-end, processing and overall speed, streamlit gave me a lot of freedom. The front-end is comprehensive and responsive with specific emphasis on UI which is a major advantage while serving for non-tech medical professionals. Here is how the home-page looks :

3. Integration

The next step was to integrate the models for real-time functioning with the streamlit front-end. I've designed the project to save time and increase efficiency all throughout. Whenever a user goes to a specific tool, he/she gets an option to upload an image, only then does the processing start. Once done, the core engine shuts down. This makes sure that we don't waste any compute.

Another important feature I designed is to maximize accuracy. Whenever there's classification between two classes in the model, the results won't be shown unless there's a difference of atleast 50%. Since we're dealing with medical cases, this is to make sure the predictions given by the model are accurate in most cases. If the above condition is not met, it simple shows an error stating the image quality was not good enough to be analyzed.

How the Tech works

The model does live inferencing from the characteristics of the image uploaded. It's first converted to a numpy array and then processed to a set dimension for a prediction. It uses the inbuilt keras functionality and openvino for final deployment. There's a custom algorithm which decides whether the model is extremely certain about the output before printing it on the front-end

Step-by-step instructions :

1) Open the app and select a tool.

2) Upload an image based on the tool on the command bar file uploader. Wait for it to load and appear on the screen. You can also resize the image as per requirements once it loads. If you don't have access to medical images, check out the datasets folder in the GitHub repository above.

3) Click the desired button to carry out analysis. Wait for the results to appear below the image. Hoorah! You just did a comprehensive medical image analysis with a single click.

Again, code, sample datasets and models are available on My GitHub Repository.

Video Demo

Complete demo with real images :

What makes it different?

There are very few tools that have put together comprehensive medical analysis with a friendly UI for medical professionals.

The ultimate ambition with OneMed is to build out these tools and make it a one-stop solutions for medical professionals of different kinds to carry out deep learning inferencing and analysis.

For future use-cases, direct integration with medical hardware and devices is possible which allows better quality computer vision analysis and faster results.

My Hardware and Code

The entire project was made on my HP Laptop with a Intel Core i5 CPU 8th gen. As I trained my models on the cloud, I didn't have to worry about on-machine RAM and GPU strength.

The entire code is available in the app.py file in my Github Repository :

https://github.com/aryansharma-prime/medstack

Thank You

I hope you enjoyed going through my project. My purpose behind this effort was to allow more people to use this technology and build upon it. Everything on my repository is open to use for anybody who might find it interesting.

Thank You :)