PACTra (Patient ACtivity Tracker)

The Problem

As the COVID-19 hit, the pandemic exposed the need for the rapid adoption of increasingly innovative digital health technologies. The telehealth is one of the key component which has received much attention. The Remote Patient Monitoring (RPM) has been gaining traction since the beginning of the pandemic. There are many benefits of RPM. It enables ease of access to patient data, the ability to deliver higher-quality care to more patients with a lower risk of burnout, lower costs and higher efficiency. RPM is even reduces the risk of spreading infection in the situation like this pandemic. RPM was still struggling to get off the ground, despite the technology being available for years. We are still in need for highly efficient devices and communication infrastructure which enables it.

The Solution I am building

I am going to build a device with Machine Learning (ML) capabilities and a system which will help doctors to monitor activities of their patients and vitals during those activities such that whether a patient is walking, sleeping/resting, exercising, or eating and what are the vitals during these activities such as body temperature, oxygen level and heart rate. This is one aspect of RPM.

How does my solution work?

I am going to build an IoT device with ML to track patients vitals during different activities which are performed by patients. My solution can tell a doctor what is the body temperature, heart rate and oxygen level of patient when he walks, sleeps, eats or exercise. Doctor can keep track of these vitals during different activities because for each activity vitals may be different. So by analyzing doctor can prescribe treatment or give suggestion to patient. If doctor advices a patient to do some exercise regularly at particular time, he can also keep track whether that patient is doing exercise as advised regularly or not using my device and if patient doesn't then he can also alert patient. In this way the solution is useful for both. The doctors will know whether patient is giving proper rest to his/her body, exercising properly if advised and having meals at proper time and patients will get alerts from doctors if they don't do such thus can improve their health.

My device will collect patients data using the sensors attached to nRF5340 DK. The sensors which are attached to DK are accelerometer, microphone, Pulse Oximeter, Heart Rate Sensor, and temperature Sensor.

The function of the sensors are given below:

Accelerometer sensor enables to monitor patient activities such as walking, sleeping/resting, exercising, or eating. A ML model is build to identify these activities.

Microphone is used to identify the patients respiratory health with the help of ML model. The study Artificial intelligence model detects asymptomatic Covid-19 infections through cellphone-recorded coughs by MIT and research paper COVID-19 Artificial Intelligence Diagnosis Using Only Cough Recordings shows that this technique is very useful in detecting the infections.

The Pulse Oximeter, Heart Rate Sensor, and temperature Sensor are programmatically aligned with the accelerometer + microphone sensor’s ML model (the model has bee exported as library and then the nRF Connect SDK will be used with the ML model code to write sensor data acquisition and transmission over BLE) to report different reading levels at different activities, such that while patients is exercising, the temperature and heart rate and pulse data may be different than while patient is sleeping. It enables to conduct study on different aspect of patient’s activities. The following figure shows the different components of the sensor node.

The sensor does capture oxygen level, heart rate and temperature of patients during different physical activities such as while patient is sleeping, walking, exercising, running etc. this helps doctors to study different health aspect of patients. The sensor node also monitor respiratory health of patient as it breath during different activities using microphone and ML model. The sensor node also detects cough.

The data from sensor node will be sent to gateway node via bluetooth or zigbee. The gateway device is locally installed at patient’s end and it displays the different activity data to patient as well. The gateway device than forwards the data to IoT cloud via MQTT which can be accessed by doctors through client Application. The following figure illustrates it.

Steps to build the project

Step 1

Attach the X-NUCLEO-IKS02A1 shield to nRF5340 DK and download the edge impulse SDK from https://github.com/edgeimpulse/firmware-nrf52840-5340-dk. To build the firmware you need to have following tools installed on your system.

• nRF Connect SDK
• GNU ARM Embedded Toolchain 9-2019-q4-major.
• nRF Command Line Tools

(note: the EdgeImpulse SDK requires nRF Connect SDK version v1.4.99-dev. Therefore make sure that you have the correct version installed)

the firmware can be build using following command

$ west build -b nrf5340dk_nrf5340_cpuapp

Now before I build a machine learning model I need to flash the firmware to nRF5340 DK. I used programmer tool that comes with nRF Connect SDK other option I can use is just drag and drop bin file to JLink folder which appears when nRF5340 DK is connected to PC via USB cabel.

Step 2

Next step is to build the machine learning model using edgeimpulse studio. Since the device is ready to get connected to edgeimpulse studio, I have connected it as shown below.

Now I can capture the data and build the model.

after capturing the data, I have created the model for patients activities as shown in following figures.

After building the model the next step is to train the model.

Now I can see the details of the Neural Network which has been built.

After this I have tested the model on live data.

Finally I can export the model. There are various options to export the model. Export as library which can be used in nRF Connect SDK to write the final code with other sensors such as pulse oximeter, heart rate and temperature sensor or download the binary to test.

Step 3

Now the other sensors can be connected and I can use the library which we have built in the step 2. I can program the nRF 5340 using the library which I have built and using the BLE library and I2C library provided with nRF Connect SDK.

Step 4

Now I need to build and program the gateway node. I am using the Raspberry Pi for my gateway node. It has built in BLE so that it can easily detect my sensor node built using nRF5340 over BLE. With the help of python script I can access the BLE data. My gateway node displays the information to user locally and it also transmit the data over IoT server via MQTT. My gateway node also keeps the most recent data in its local database which is not sent to IoT server to make sure that the server always have all the data. Once the data to the server is sent, the gateway node can delete that data to make more space for new data as the gateway node itself is a resource constraint device. It only keeps that data in local database which has to be displayed to the user locally and which is not sent to the server.

Step 5

In this step the final server application is built which can store data and display it on user interface using graphs and other GUI elements. Here the data is permanently stored. This web application can also be accessed by patients and doctors with credentials on laptop or smartphones.

Conclusion

The nRF5340 is one of the best options available to build the wearable devices as this board is powerful to run ML models and can be operated using even a coin cell which is a perfect choice to build the advance medical device like the proposed system. The nRF Connect SDK is a huge collection of objects for different sensors and hardware. It provides well defined ready to use code for different purpose such as reading data from different sensors and transmitting the data using BLE, Zigbee, Thread or other protocols over a short or long range. The power profiler kit can also be used to measure the power performance of the sensor node. The proposed project is based on Internet of Things (IoT) which harness the power of nRF5340 and nRF Connect SDK to build a smart patient monitoring system which is an essential need today. The proposed system is also flexible and extensible as new sensors can be easily added and configured. The system uses MQTT protocol for communication over the Internet which gives fast and reliable performance.