Snore Smart Detection

Introduction:

According to sleep foundation, snoring affects over 90 million American adults, 37 million on a regular basis. That's more than a quarter of entire US population.

Snoring can create real problems in a marriage. Also obstructive sleep apnea (when breathing is disrupted during sleep for short periods), which increases the risk of developing heart disease.

Sleep apnea is manageable using several approaches including CPAP (continuous positive airway pressure), oral appliance therapy and surgery.

image from mayo clinic

Using AI to solve the issue

With advancement of AI, where voice recognition can easily pick up different people's voice when calling "Alexa", we can use the same technology to recognize different types of snoring, and send the information back to user's phone with the recordings.

Our solution is to use audio classification through deep learning to identify the type of snoring which can be highly useful for a targeted and thus successful medical treatment V – Velum (palate), including soft palate, uvula, lateral velopharyngeal walls O – Oropharyngeal lateral walls, including palatine tonsils T – Tongue, including tongue base and airway posterior to the tongue base E – Epiglottis.

Demo

1. Setup Sparkfun Redboard Artemis ATP

First clone/fork this repository:

https://github.com/coolioxlr/ArtemisATPTensorflowMicroSpeech

This repository contains the Micro Speech example that demonstrates the use of TensorFlow Lite based machine learning executing on the SparkFun Artemis ATP Development. The example is designed for use within the Arduino development environment.

The example contained in this repository, micro_speech, uses the on-board PDM microphone to detect the kind of snore.

This repository is a copy of sparkfun/Tensorflow_AIOT2019 repository adapted for the SparkFun Artemis ATP Development

Contents

• Required Hardware
• Software Setup
• Running an Example

Required Hardware

To run the examples, the following hardware is required:

• SparkFun Artemis ATP Development
• Reversible USB-A to USB-C cable or a cable to connect the Development Board to the development computer.

Software Setup

The example in this repository is for use and execution within the Arduino development environment. This section details the steps required to setup Arduino for the examples.

Install Arduino

Arduino is available for a variety of platforms. To ensure compatibility with the demos in this repository, the latest version should be installed.

The Arduino application is available for a variety of platforms and is available online from Arduino. Download the application from the Arduino website using this link.

Install the TensorFlowLite Library

The examples utilize the TensorFlowLight Arduino library, which is installed using the Arduino Library Manager.

To install this library, use the following steps:

• In Arduino, select the "Manage Libraries..." menu item. Tools > Manage Libraries...
• In the Library Manager interface, search for tensorflow
• Select the library Arduino_TensorFlowLite by TensorFlow Authors
• Select the non-precompiled version of the library
• Select the Install button to install the library

Install the SparkFun Boards Package

Load the SparkFun Boards package into the Arduino Board Manger.

To install package, use the following steps:

In Arduino, open the Preferences menu item. File > Preferences, (macOS) Arduino > Preferences

• In Arduino, open the Preferences menu item. File > Preferences, (macOS) Arduino > Preferences

Add the following path to the Additional Boards Manager URLs: path in preferences.

https://raw.githubusercontent.com/sparkfun/Arduino_Boards/master/IDE_Board_Manager/package_sparkfun_index.json

• Add the following path to the Additional Boards Manager URLs: path in preferences.https://raw.githubusercontent.com/sparkfun/Arduino_Boards/master/IDE_Board_Manager/package_sparkfun_index.json

Select the OK button to save the preferences.

• Select the OK button to save the preferences.

Install the SparkFun Artemis Boards Package

Once the location of the SparkFun boards package is set in the preferences, the board definition package for the SparkFun Apollo3 boards must be installed.

To install package, use the following steps:

• In Arduino, open the Preferences menu item. Tools > Board "..." > Manage Boards...
• Search for SparkFun
• Select the **SparkFun Apollo3 Boards" package
• Close the dialog
• Select the SparkFun Edge Board. Tools > Boards "..." > SparkFun RedBoard Artemis ATP
• Set the Bootloader to the SparkFun Variable Loader. ***Tools > Bootloader "..." > SparkFun Variable Loader..."

Running an Example

Configure the Hardware

• Plug USB cable into computer
• Plug in Artemis ATP board to adapter

Configure the Software

• Install and setup Arduino as described above.
• Download (or clone) the repo to favorite location

https://github.com/javagoza/ArtemisATPTensorflowMicroSpeech.git

• Open Arduino

Open an example from this repo through Arduino by clicking on a .ino file in one of the example directories

• micro_speech
• Open an example from this repo through Arduino by clicking on a .ino file in one of the example directoriesmicro_speech
• Select the SparkFun RedBoard Artemis ATP board (Tools->Board under 'SparkFun Apollo3')

Select the proper port to connect with the board (Tools->Port)

• Windows - COM Port
• Linux - /dev/ttyUSB*, where * is a number. NOTE: Permissions on this device needs to include user read and write (chmod 666)
• macOS - /dev/cu.usbserial*
• Select the proper port to connect with the baord (Tools->Port)Windows - COM PortLinux - /dev/ttyUSB*, where * is a number. NOTE: Permissions on this device needs to include user read and write (chmod 666)macOS - /dev/cu.usbserial*

Change the Bootloader from Ambiq Secure Bootloader (Default) to SparkFun Variable Loader (Enable w/ Artemis Bootloader) (Tools->Bootloader)

• (Your boards have received the upgraded bootloader already)
• Change the Bootloader from Ambiq Secure Bootloader (Default) to SparkFun Variable Loader (Enable w/ Artemis Bootloader) (Tools->Bootloader)(Your boards have received the upgraded bootloader already)
• Change the SVL Baud Rate from 921600 to 230400

Upload

• Compile with the Verify button (checkmark symbol)

Upload with the Upload button (arrow symbol)

• If uploading fails try lowering the bootloader baud rate
• Upload with the Upload button (arrow symbol)If uploading fails try lowering the bootloader baud rate
• Note: You will see some warnings from the compiler, these can be ignored.
• Note: The bootloader attempts to upload for 3 times. If all three attempts fail, try adjusting the baud rate.

Serial Monitor

• Open the Serial Monitor by clicking on the magnifying glass icon (top right) or (Tools->Serial Monitor)

2. Training a Snore Model

An example of how to train a new model based on Google collected sample phrases is included in the tensorflow micro_speech example repository. The method listed utilizes Google Colaboratory to run the training session - an Juypter notebooks based system that presents a Python based notebook and abstracts the management of computing resources.

The notebook to run the training for this example is contained in the tensorflow github repository at:

https://github.com/tensorflow/tensorflow/blob/59c06b9016700dbf1ab0cefc062d247345cdd0f0/tensorflow/lite/micro/examples/micro_speech/train_speech_model.ipynb

We are going to switch out the dataset to our snore dataset.

Also change the wanted word to the following

Configure training

The following os.environ lines can be customized to set the words that will be trained for, and the steps and learning rate of the training. The default values will result in the same model that is used in the micro_speech example. Run the cell to set the configuration:

Install dependencies

Next, we'll install a GPU build of TensorFlow, so we can use GPU acceleration for training.

We'll also clone the TensorFlow repository, which contains the scripts that train and freeze the model.

Load TensorBoard

Now, set up TensorBoard so that we can graph our accuracy and loss as training proceeds.

Begin training

Next, run the following script to begin training. The script will first download the training data:

Freeze the graph

Once training is complete, run the following cell to freeze the graph.

Convert the model

Run this cell to use the TensorFlow Lite converter to convert the frozen graph into the TensorFlow Lite format, fully quantized for use with embedded devices.

Finally, we use xxd to transform the model into a source file that can be included in a C++ project and loaded by TensorFlow Lite for Microcontrollers.

Testing

We are going to test the output accuracy.

Transfer the new compiled model to Redboard Artemis ATP

After the above steps, we generated the following.

Open the model data used is in the file:

micro_features_tiny_conv_micro_features_model_data.cpp

And replace the following lines https://github.com/javagoza/ArtemisATPTensorflowMicroSpeech/blob/master/micro_speech/micro_features_tiny_conv_micro_features_model_data.cpp#L34 with the text in tiny_conv.cc

micro_features_micro_model_settings.h with the following

micro_features_micro_model_settings.h

change constexpr int kCategoryCount = 6;

Compile and Upload to Redboard Artemis ATP and test the snoring.

What's Next

Connect to other wearables e.g. watch that will nudge you when you snore.