No Pause

Overview

A hot flash is the sudden feeling of warmth in the upper body, which is usually most intense over the face, neck and chest. Although other medical conditions can cause them, hot flashes most commonly are due to menopause. In this project, I build a device which can be used to detect hot flashes and trigger some action to relieve the person, in this case turn on Air Conditioner cooling system using IR emitter. As its input, it takes multidimensional infrared thermal sensor data. Its output will be a simple classification that notifies us if person is recognized and there is a sudden change in temperature has recently occurred.

Hardware consideration

I wanted to make a wearable device which can read body temperature data. After doing some research and implementations I found wearing sensors all the time especially during sleeping is uncomfortable and not feasible in many cases. A hot flash can also cause sweating and wearing a battery powered device needs special housing to accurately measure data. Also, for cooling I investigated thermoelectric Peltier cooler modules and found most of them needs bulkier 12V battery to operate properly for optimal cooling and it is also unsafe if it is not controlled carefully and can cause burns. For a non-invasive solution, I found low resolution thermal camera can read temperature reliably from a distance and they can work in dark. Although the reading accuracy may be affected by distance but calculating continuous or sudden change in the thermal image data can be recorded accurately. I chose Atom Matrix as a microcontroller device because it is small, cheap, can run TensorFlow model and has an IR emitter which can be used as a remote control. I chose Wio Terminal to capture thermal camera data for training because it has an LCD screen with many buttons which is used to capture data for different categories.

Data collection for training

The first and the most important step in a machine learning project is to collect the training data in such a way that it should cover most of the representative cases for a given recognition task. The 3 buttons on the Wio Terminal was used to label the 3 classes.

The captured data is saved to the files on a micro SD card attached to the Wio Terminal. Each thermal image data was captured as a separate file. The file contains no header line, only the comma-separated 768 (24x32) temperature readings. An example readings file contents looks like as follows.

26.47,25.97,25.85,25.72,26.90,26.12,26.60,26.86,27.00,26.68,26.90,26.74,27.78,27.21,27.75,29.12,31.29,31.50,32.24,31.95,31.72,30.80,31.29,30.69,31.18,30.86,31.46,31.37,29.21,28.23,28.18,28.03,25.83,26.33,25.55,26.56,26.59,26.90,26.52,27.38,26.94,27.39,26.85,27.21,27.32,27.66,28.81,30.45,30.97,31.74,31.55,32.14,31.37,31.03,30.63,30.69,31.03,31.52,30.85,31.14,28.80,28.57,27.81,28.39,26.43,26.24,26.67,26.71,27.13,26.99,27.63,28.07,28.59,28.39,27.80,28.19,28.11,28.25,30.91,32.15,31.78,31.46,31.82,31.33,31.10,30.43,30.37,30.06,29.77,29.84,30.58,30.45,29.28,28.42,28.34,27.76,26.31,26.62,26.38,27.24,27.27,27.91,28.94,29.11,30.11,29.73,30.25,29.53,29.59,29.22,32.01,32.70,33.17,32.00,31.15,31.52,30.59,30.46,29.87,30.07,29.43,30.09,30.01,30.68,29.10,28.91,27.99,28.34,26.59,26.60,26.99,27.49,28.68,29.64,31.88,33.14,33.41,33.02,32.48,32.83,32.60,32.60,33.58,34.16,34.79,34.58,32.43,32.15,31.07,30.77,29.84,29.89,30.01,29.48,29.99,29.63,29.33,28.47,28.23,27.90,26.26,26.26,27.13,28.21,30.50,31.41,33.23,33.70,33.44,33.40,33.38,33.18,33.31,33.12,33.65,34.33,34.81,34.93,33.96,32.50,31.29,30.82,29.37,29.93,29.13,29.93,29.29,30.07,28.76,29.00,28.38,28.69,26.83,26.55,27.36,28.68,32.50,33.12,33.78,33.40,34.17,34.14,33.80,33.56,33.84,33.35,33.85,33.54,34.63,34.45,34.68,34.49,31.76,30.94,29.87,29.38,29.67,29.20,29.45,29.68,29.02,28.42,28.63,28.49,26.29,27.30,27.51,28.71,31.96,33.70,33.86,33.76,33.87,34.32,33.60,33.94,33.41,33.39,33.74,34.04,34.32,34.95,34.24,34.63,31.81,31.15,29.59,29.78,29.19,29.62,28.77,29.72,28.88,29.12,28.51,28.79,26.45,27.15,28.06,28.72,32.45,33.05,33.89,33.84,33.60,33.39,34.02,33.69,33.64,33.29,34.11,33.81,34.68,34.55,34.75,34.15,32.55,31.42,30.25,29.76,29.56,29.42,29.64,28.59,28.87,28.66,29.10,28.79,26.76,27.06,27.39,28.97,31.63,33.46,33.78,34.22,33.97,34.11,33.58,34.20,34.01,33.97,33.69,34.16,34.54,34.90,34.33,34.63,33.09,31.81,29.81,30.15,28.89,29.81,28.96,29.39,28.70,29.35,28.54,29.07,27.18,26.59,27.31,27.98,31.52,32.56,33.37,33.52,33.72,33.72,33.52,33.22,33.75,33.27,33.93,34.13,34.69,34.65,34.26,34.12,32.70,31.12,30.58,30.45,29.91,29.42,29.72,28.82,29.66,29.10,28.91,29.04,26.26,26.72,26.66,27.70,29.47,31.74,32.43,33.58,33.48,33.45,32.63,32.79,31.94,33.32,33.36,34.14,34.42,34.55,33.99,33.87,31.15,30.99,30.15,30.84,29.81,30.02,29.24,29.61,29.15,29.18,28.82,29.32,27.21,26.60,27.01,26.83,27.83,27.64,29.27,29.08,30.97,29.96,29.96,28.30,29.66,30.77,34.03,33.60,34.23,33.26,32.36,31.42,30.61,30.63,30.44,30.50,30.45,30.46,29.70,29.37,29.70,29.34,29.17,28.83,26.28,26.53,26.36,27.31,26.68,27.55,27.46,28.27,27.83,28.53,27.72,28.01,28.14,30.84,32.27,33.38,32.09,32.70,31.33,31.32,30.03,30.53,29.86,30.86,30.35,30.88,29.78,29.66,29.58,29.62,29.06,29.77,26.63,26.16,26.65,26.85,27.10,26.79,27.12,26.67,27.47,27.14,27.15,27.15,27.93,28.86,31.42,31.62,31.87,31.27,31.11,30.65,30.47,30.46,30.62,30.17,30.45,29.94,29.73,29.38,29.33,29.33,29.18,29.06,25.92,26.74,26.20,26.93,27.03,26.89,26.71,26.92,26.90,27.08,26.74,27.46,27.06,28.62,31.36,32.15,31.96,31.97,31.03,31.01,30.20,30.71,31.08,31.04,30.51,30.21,29.31,29.81,29.18,29.43,29.07,29.56,26.80,26.61,26.85,26.61,26.86,26.81,26.86,27.13,27.21,26.94,26.84,26.77,27.24,26.68,30.82,31.45,32.56,31.93,31.30,31.13,31.15,31.29,31.83,31.84,31.02,30.16,29.38,29.19,29.56,29.13,29.41,28.83,26.54,26.56,26.58,26.95,26.81,27.11,26.44,27.07,26.81,27.04,26.77,27.05,26.85,26.94,28.29,30.73,31.36,31.84,30.73,30.88,30.73,30.90,31.61,31.92,30.55,30.41,28.76,29.27,29.02,29.64,29.32,29.59,26.79,26.32,26.78,26.93,26.85,26.80,26.95,27.14,26.90,26.59,26.60,26.76,27.19,27.05,27.35,27.16,28.37,28.20,29.84,29.51,31.84,32.69,32.54,31.55,31.03,30.36,29.18,28.63,29.54,28.92,29.18,29.26,26.28,26.71,26.30,27.09,26.60,26.93,26.71,27.05,26.46,27.39,26.90,27.22,26.64,27.33,26.61,27.39,27.15,28.14,28.76,29.57,31.26,32.76,31.79,31.63,30.77,31.15,29.09,29.45,28.78,28.99,29.07,29.83,26.96,26.68,26.70,26.47,26.79,26.53,26.86,26.56,27.08,26.60,26.86,26.50,27.11,26.74,27.29,27.29,27.16,27.50,28.66,28.47,30.35,30.02,30.71,30.36,31.65,31.08,30.13,29.60,29.28,29.08,29.37,29.02,26.45,26.86,26.63,27.07,26.53,27.05,26.85,27.21,26.48,27.28,26.80,27.12,26.54,27.20,26.52,27.19,26.77,27.61,27.91,28.58,28.91,29.42,29.19,30.28,30.07,31.03,30.25,29.85,29.17,29.61,29.30,29.79,26.88,26.71,26.97,26.73,27.01,26.79,26.83,26.99,27.51,26.93,26.96,26.79,27.22,26.88,27.34,26.87,27.20,27.23,27.70,27.45,28.61,28.18,29.45,29.06,29.99,29.77,30.61,29.50,29.72,29.00,29.41,30.15,25.94,27.25,26.77,26.97,26.82,27.24,26.67,27.18,26.75,27.16,27.01,27.37,26.91,27.49,26.81,27.24,26.89,27.97,27.36,28.21,27.73,28.77,28.52,28.88,29.20,30.00,29.95,30.37,29.49,29.48,28.84,29.91

The visual representation of the captured data is as follows:

An Arduino sketch (Thermal_camera_data_collection.ino) to capture training data is available at the Github repository mentioned in the code section.

Data collection session video

Data splits for training and validation

The data were captured for 3 categories: person (in different orientations and postures), hot or cold objects, and background (nearly uniform temperature for example, wall or empty room). A little over 100 samples for each classes were captured. The collected data has been split into training (60%), validation (20%), and testing (20%) datasets. Since the data was collected from the infrared temperature camera (MLX90640) which is already calibrated and they are already within a specified range so we can use the raw data as is for training and inferencing.

Model Architecture

We can think of the input data as an image of 24x36 pixels. A convolutional neural network is one of the best options suited for recognizing patterns in images and time-series sequence data. The first few layers are 2D convolution neural networks with few other regularization layers. The last layer is a fully connected dense layer with softmax activation which outputs a probability of all 3 classes. The summary of the model is given below.

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 24, 32, 8)         80        
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 24, 32, 8)         584       
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 12, 16, 8)         0         
_________________________________________________________________
dropout (Dropout)            (None, 12, 16, 8)         0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 12, 16, 8)         584       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 6, 8, 8)           0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 6, 8, 8)           0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 6, 8, 16)          1168      
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 3, 4, 16)          0         
_________________________________________________________________
dropout_2 (Dropout)          (None, 3, 4, 16)          0         
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 3, 4, 16)          2320      
_________________________________________________________________
flatten (Flatten)            (None, 192)               0         
_________________________________________________________________
dense (Dense)                (None, 64)                12352     
_________________________________________________________________
dropout_3 (Dropout)          (None, 64)                0         
_________________________________________________________________
dense_1 (Dense)              (None, 32)                2080      
_________________________________________________________________
dropout_4 (Dropout)          (None, 32)                0         
_________________________________________________________________
dense_2 (Dense)              (None, 3)                 99        
=================================================================
Total params: 19,267
Trainable params: 19,267
Non-trainable params: 0

Model Training and Evaluation

The training of the model was done on an Intel NUC with Linux and an eGPU (NVIDIA GTX 1080Ti). Although it just takes couples of minutes to train on a CPU but the development process becomes pretty slow while testing out different architectures and hyper-parameters. The TensorFlow 2.1 with Keras API is used for model creation and training process. I created a Jupyter notebook for data processing, training and the final model conversion. All code are available at Github repository which is mentioned in the code section. The training accuracy is 99% and evaluation accuracy on test data is 92.86% which can be further improved with more training datasets and model hyper-parameters tuning.

Inferencing on the device

The created model is converted to the TensorFlow Lite model and the converted model is transformed into a C array file for deploying with the inferencing code. The TensorFlow Lite Micro SDK is used to run inference on the device. I have created an Arduino sketch (Hot_flash_detector.ino available at Github repository) for inferencing and displaying the result. The microcontroller receives the samples continuously from the thermal camera sensor at the interval of 500ms and detect the sudden change in the temperature of the recognized person based on last 20 average maximum temperature readings.

Inferencing demo

The use cases for common benefit and cost

It is an easy to use low-powered device which can run on battery for weeks. It can be used safely for women facing hot flashes problem during day or night. This device is also low cost. The total cost of the final working product (Thermal camera + Atom matrix) is well below 50 USD and can be further reduced if it is mass produced.

Scope for improvement

The device can be used to detect sleeping problem by analyzing thermal images time-series data and can trigger a music system to play some soothing music or control smart lighting systems. Also, some analytics data can be saved locally at the Atom Matrix's SPI flash memory and can be synced using a mobile phone app over BLE for further analysis.