Fridge Doctor : Refrigerator Predictive Maintenance

Refrigerator is one of the most important and very useful household appliance. We use it every day for storage and preservation of food, vegetables, drinks, medicines etc. But If this appliance fails abruptly due to some fault, it spoils the food/vegetables, medicines or reduce their effectiveness. Considering the pivotal importance of a refrigerator in our lives, In this project, I will try to make system to monitor behavior of refrigerator parameters (Such as cooling systems - Temperature, Humidity, compressor Sounds/vibrations, Internal Gasses etc.) to identify a refrigerator fault/failure allowing us to predict its maintenance.

So in order to monitor the Fridge's cooling system and Compressor's pivotal parameters we must need

1. Good quality sensors - Temperature and Humidity,

2. IMU sensors - Gyroscope, Accelerometer

My proposed solution is to use a machine learning (ML) model to identify a failure as soon as possible using the temperature and humidity changes in a refrigerator.

Therefore we also need a good development board/MCU board/ which can run ML models and should have wireless capability. I came to know about this "Thingy:53" development board and the contest "Smarter Sustainable World Challenge with Nordic Semiconductor" on Hackster, and I decided to apply for this interesting contest.

Thank you #Hackster and #NordicSemiconductors !

Thank you very much @Hackster @NordicSemiconductors and other Contest Supporters for organizing such amazing contest, "Smarter Sustainable World Challenge 2022" and sponsoring and supporting my Idea with "Nordic Thingy:53 Prototype Kit".

Nordic Thingy:53

The Nordic Thingy:53 is a multi-sensor prototyping platform for Matter, embedded machine learning (ML) and other wireless IoT products. It is based on the nRF5340 System-on-Chip (SoC), our current flagship dual-core wireless SoC. With integrated sensors for motion, sound, light and environmental factors, it is the perfect platform for building proof-of-concepts and developing new prototypes in a very short time.

The features which we desired to implement our above mentioned solution are comes pre-built with this amazing device

Multisensory - Thingy:53 also includes many different integrated sensors, like environmental, and color and light sensors, accelerometers, and a magnetometer, which all can be taken advantage of without additional hardware. It is powered by a rechargeable Li-Po battery with a 1350 mAh capacity that can be charged via USB-C. There is also an external 4-pin JST connector compatible with the Stemma/Qwiic/Grove standards for hardware accessories.

WirelessCapability - The Bluetooth Low Energy (LE) radio feature allows updating Thingy:53's firmware and communication over Bluetooth LE and the radio also supports other protocols like Bluetooth mesh, Thread, Zigbee, and proprietary 2.4 GHz protocols.

Machine Learning - The processing power and memory size of its dual Arm Cortex-M33 processors enables it to run embedded machine learning (ML) models directly on the device. Every Thingy:53 comes pre-installed with the firmware to work with the nRF Edge Impulse app

Getting Started with Thingy:53

To get started, please follow the Nordic Thingy:53 Getting started guide.

To develop applications for the Thingy:53, We can use the nRF Connect SDK, which contains a variety of compatible libraries and samples. Nordic also have a dedicated guide for development on Thingy:53, see Developing with Thingy:53.

Nordic's Dev Academy has detailed course about how to get started with developing applications with nRF Connect SDK and Visual Studio Code Integration. We will also need nRF Connect for Desktop app for that.

The Nordic Thingy:53 is preloaded with Edge Impulse application firmware, MCUboot bootloader, and a SoftDevice controller on the network core of the nRF5340 SoC. For more information on the firmware and associated features, see Developing with Thingy:53.

In addition to Edge Impulse, the nRF Connect SDK contains multiple precompiled samples that we can use to program to the Nordic Thingy:53. For a non-exhaustive list, see Samples and applications compatible with Thingy:53 in Developing with Thingy:53.

To try different samples or to update the existing firmware on the Nordic Thingy:53, we can use following different methods,

1. Via Bluetooth® Low Energy (LE) using nRF Connect App/Programmer App

2. USB (MCUboot), or

3. External debug probe.

You can know more about Thingy:53 on Nordic Semiconductors website.

Now let me explain how is my proposed solution will going to work.

Project Concept

Above figure shows functional block diagram of the project. We will keep Thingy:53 in bottom compartment of Fridge, nearer to compressor (so we can sense its abnormal vibrations due to overload/fault). The Fridge which we have used in this project have its compressor on back side (of bottom compartment). as shown in below.

With the help of on-board sensors of Thingy:53, we can sense the temperature and humidity changes of refrigerator, during its cooling cycles. Also Gyroscope and Accelerometer data can also help us to identify when compressor gets overload and shakes abruptly. By analyzing those metrics with a machine learning algorithm, it can then be possible to identify any abnormal operations that would be indicative of a failure, like a compressor that has gone on the fritz, or a dirty coil etc.

Dataset - A good machine learning model starts with a good dataset. Our machine learning model needs at least two kinds of data to identify refrigerator failure.

• Normal operation data - Time-series data from a normally working refrigerator.
• Abnormal operation data - Time-series data from a faulty refrigerator.

Different kinds of faults may generate a different pattern in data. For example, a non-working compressor will never decrease the temperature when it stops working. In contrast, a clogged or dirty coil may force the compressor to work harder than usual, taking more time to achieve the target temperature. Also Faulty compressor causes fluctuations and abrupt vibrations when at fault or load.

Unfortunately, I do not have access to a faulty refrigerator for data collection; therefore, I have simulated "abnormal operation" data by - "Keeping the fridge door open for an extended period". This event should increase the temperature and hopefully force the compressor to work harder, thus simulating a fault state.

To prepare the dataset we need to capture -

• Temperature,
• Humidity,
• IMU data - Accelerometer and Magnetometer

Getting started with machine learning

I used Edge Impulse Studio for Data Acquisition and Dataset Preparation. As mentioned earlier, The Nordic Thingy:53 is preprogrammed with Edge Impulse firmware. To connect the Nordic Thingy:53 to the Edge Impulse Studio, we can use the nRF Edge Impulse mobile application to connect over Bluetooth LE, or connect the Nordic Thingy:53 to a computer to connect over USB.

I've also used nRF connect SDK V2.1.0 and Visual Studio Code to build the pre-compiled sample of nRF Edge Impulse Firmware. Then using DFU MCUBoot method uploaded that firmware into Thingy:53

The Edge Impulse firmware enables data collection from all the sensors on the Nordic Thingy:53. You can use the collected data to train and test machine learning models. Deploy the trained machine learning model to the Nordic Thingy:53 over Bluetooth LE or USB. Here is the complete guide and steps to get started with Edge Impulse.

I used data forwarder tool of Edge Impulse to collect the "Environment and IMU data" from Thingy:53. For that first

1. I flashed/updated latest Thingy:53 Edge Impulse firmware using this guide. you can also use nRF Programmer App

2. Then by logging into my Edge Impulse Account I created new project.

3. Then I downloaded and installed nRF Edge Impulse app on my Android Mobile from PlayStore.

4. Then after turning on Nordic Thingy:53, kept it in Fridge.

5. Then I connected with Thingy:53 over BLE using my nRF Edge Impulse app from my Mobile.

4. Using this (nRF Edge Impulse app), Now we can "Wirelessly" collect samples of data for Each labels,

• Normal Operation (Fridge door Closed)
• Abnormal Operation (Fridge door open for longer duration-Simulated Fault)

Data Classes -

As mentioned previously, due to the unavailability of a faulty refrigerator, I have simulated the abnormal operation using just one technique. That gives me only two classes of data - normal operation and abnormal operation. I collected total 8 hours data in total 960 samples of 30 seconds each. Which includes 4 hours as a normal operation. and 4 hours of abnormal operation.

Then I used automatic Data Split tool of Edge Impulse to spit the data into "Training data and Test Data"

Data got split as shown below

As we were considering previously, about simulated fault, means, abnormal behavior of compressor. When I kept fridge door open for extended time, compressor struggled and kept on trying to achieve the set temperature (3 to 4°C). Some erratic movements, vibrations also had been observed/recorded during that experiment. Lets see how we can use this all data to classify and detect the anomaly.

I used nRF Edge Impulse App to collect all the data. Since I've to keep Thingy:53 in Fridge and keep the door open/close. So there is no chance of wired connection. But as we can connect thingy:53 over BLE witlessly as I explained earlier, It was easier to collect the data. But we can also connect Thingy by WebUSB as shown in figure

Training -

For training ML model, I used Edge Impulse Studio (EI Studio), which is an excellent tool for building ML solutions lot quicker, also Thingy:53 comes in supported list of hardware by Edge Impulse. Edge Impulse has many excellent features for all stages of building an ML solution. One such cool feature is Data Explorer which makes visualization of the data points very easy. I've used Spectral Analysis preprocessing block of Impulse.

I used data visualization tools to assess the training data, and you can see that there appeared to be good and clear separation between the two classes, indicating that the sensors we chosen (Environment + IMU) were appropriate for detecting misbehaving refrigerators.

We can also use "Feature Explorer" tool in Edge Impulse for more detailed view of spectral analysis block output for each sample, as following.

Impulse Design - Since the primary goal is to recognize when a refrigerator is in any abnormal state, I used two different approaches to design an Impulse, with respect to learning block. I used Spectral analysis in preprocessing as I mentioned earlier, since it was recommended by Edge Impulse and It is for analyzing repetitive motion, such as data from accelerometers. Extracts the frequency and power characteristics of a signal over time. At learning stage I tried both recommendation of Edge Impulse. As shown in following Image,

So the impulse design for both approaches looked like following,

Thanks to Edge Impulse tool the data analysis pipeline can be constructed with just a few clicks, The model was then trained using the collected dataset.

Testing -

After the training process completed, the model testing tool was used to validate performance. There were 1.5 Hrs of data was split for testing purpose. Model testing uses that raw data to check how model will react to unexplored data. Overall, the (Keras based) model was found to predict operational state with excellent accuracy, no sample producing anomaly.

and Live classification results for one sample was as following,

For Impulse design with Anomaly detection (K-Means) learning block, Edge Impulse suggested 4 axes *

• accY Kurtosis
• Humidity Kurtosis
• magX Spectral Power 281.25-343.75Hz
• magY Kurtosis

Training and Testing model with same Impulse Design(Anomaly Detection K Means Learning Block) given following results

This impulse model (K-Means) failed to detect any Anomaly in live classification results. So I decided to use Keras based classification model for final deployment, which have given good results in testing as well as in live classification experiment.

Deployment -

Edge Impulse fully supports the Thingy:53 development board, so the best way to deploy this model would be to build firmware. There is also C++ library option but I'll save it for future exploration.

Model optimizations can increase on-device performance but may reduce accuracy. Edge Impulse also features "EON Compiler" which gives Available optimizations for NN Classifier.

After completing Impulse design, training, and Testing on Edge Impulse Studio. I build the firmware using above method. Then flashed resulting.zip firmware file

using nRF Connect for desktop App and Programmer v3.0.4

For wireless operation, I used nRF Edge Impulse app to Deploy and Test the Inference in following way.

Demo

Following video shows live demonstration of the project. Thingy:53 and our deployed Impulse (ML Model) successfully identifies simulated compressor fault. Hence using proposed method with the help of more accomplished/thorough dataset(covering more faults/problems), we can easily predict maintenance of the Refrigerator before its breakdown.

Future

There can be lot of factors affecting abnormal behavior of refrigerator. Clogged or burnt or dirty coil, leakages, Overheating of compressor etc. So, Testing with real faulty Refrigerator will definitely add more aspect and features for monitoring behavior of refrigerators. I would definitely like to test this project with actual faulty refrigerator. And see how the ML model performs and detect the fault.

Current implementation do not use any custom coding. I do not own any Nordic DK board or External Debugger to upload custom binary file into Thingy:53. I used only pre-compiled sample binaries to program Thingy:53 with Edge Impulse Machine Learning Firmware using USB MCUboot method only. But, Bare metal programming using nRF connect SDK, requires lot of configurations for building the custom code.(Which I desired to, like adding fridge door alarm/Food spoilage notification etc). Also if we change anything precompiled samples, code/build breaks with no clues to troubleshoot. So in future I'd also like to build my own custom firmware with Edge Impulse C++ library of my model, by carefully configuring/selecting only required modules from nRF connect SDK. and then test it on Thingy:53.

Suggestions about Project design, Circuits, code modifications, Optimization, Errors/Bugs findings and queries are most welcomed.

Credits

Thank you @Hackster, @NordicSemiconductor, @Edge Impulse and all other Contest Supporters, peoples, websites and entities who helped me (directly/indirectly) in this project.

Reference and Credits - Big thanks to Engineer Swapnil Verma for his excellent guiding project.