Detect the Drain Blockage with Tiny ML + LoRa

My office has a lovely balcony.

Stepping into this zone is like stepping into a tranquil garden. During spring and summer, the balcony is full of vibrant colors - blooming flowers and plants create a truly delightful atmosphere. But with all that beauty comes a downside – fallen petals and leaves that get swept away by strong winds or rain and can quickly clog up our balcony drain! I want to find a way to detect when there was an accumulation of debris for our balcony drain and send alerts for timely action.

To find the best fit solution, I listed out the main obstacles I am facing.

1. I don’t have any wiring skill, there is no way I can wire the sensor out in the balcony, wireless sensor is my only choice.

2. Bad Wi-Fi connection in balcony. I need a strong, reliable connection even in complex environments and long range.

3. Outdoor use, low maintenance, low power consumption, I don't want to charge the battery every day.

4. I am really bad at coding, I only want no-code solution.

5. I am just a newbie of TinyML, I don't know much about training model and neural network architecture.

Based on what are listed above, I think I need an outdoor smart image sensor that can do local inferencing and then transmit the results with LoRa, and I choose Edge Impulse to help me with the model training.

Quick intro of Seeed SenseCAP A1101 and Edge Impulse (EI) in case you need.

A1101 is a smart image sensor, supporting a variety of AI models such as image recognition, people counting, target detection etc.

Equipped with IP66 enclosure and industrial-grade design, it can be used in challenging conditions such as places with extreme temperatures or difficult accessibility. It combines TinyML and LoRaWAN® to enable local inferencing and long-distance transmission which are the two major requirements from outdoor use.

What's more, it is battery powered. This means that data collected by the device can be transmitted without needing an external power source, making it ideal for remote monitoring applications.

Edge Impulse is the leading development platform for machine learning on edge devices, it fully supports A1101. For me, it is a no code one-stop platform, and I can sample raw data, build models, and deploy trained machine learning models to A1101 directly from the EI studio without any programming required.

Notes: A1101 is equipped with LoRaWAN connectivity, please make sure there is LoRaWAN network coverage on site. Otherwise, you will need to deploy a gateway. Click here to check LoRaWAN gateway.

Let’s get started!!

Step 1 Install EI CLI

Please refer to this documentation from EI to install Python 3, Node.js and EI CLI.

Some tips I would like to share:

1. I ran into the following problem when I install the CLI - it means no Xcode or CLT version is detected. So, I reinstalled it by using xcode-select --install.

2. You may encounter permission error such as “Error: ACCESS: permission denied”, you can solve this by adding sudo at the beginning of the command. For example, to install the Edge Impulse CLI with elevated privileges, input-

sudo npm install -g edge-impulse-cli –-force

3.Quick check to see if you have installed the CLI successfully. If you type ed+ tab, all the commands will be listed out as below.

Step 2 Connect A1101 to EI studio

1. Connect A1101 with your laptop with a Type C cable.

2. Run edge-impulse-daemon in terminal (I am using MacOS Monterey V12.5.1), then you will find A1101 is listed under your Device with Green Light, meaning that it is connected successfully with EI’s remote management API.

Step 3: Gear up for Data Acquisition

To ensure that the output model is the best fit for the A1101, I used the A1101 device to capture the samples instead of taking pictures with my phone or other cameras. This approach helped to ensure that the samples were captured under the same conditions and with the same hardware as the final deployment, resulting in a more accurate and reliable model.

A funny little interlude: the rain decided to make an appearance during the sampling. Thankfully, A1101 and I were both waterproof, and apparently my laptop was more important than me, as it was the only one that got to stay dry under an umbrella.

Step 4: Start sampling

I collected some petals and leaves to simulate different level of blockage.

I categorize all the pictures into 3 names, clean, little blocked, blocked.

Input name, set the camera into 160x160 resolution to have a clearer picture for labeling. Then click Start sampling.

I collected 196 pictures, 52 clean, 67 little blocked, 77 blocked. Since the clean feature is easily recognizable, I collected fewer samples of it compared to the other two features.

Step 5: Labeling

Click Labeling queue.

Labeling pictures is a repetitive task. Labeling 10 pictures requires dragging and moving a box and labeling 10 times, while labeling 100 pictures requires doing it 100 times. While doing this by myself, I have a deep respect for people who contribute to all these open data labeling sets, there is way too much invisible work behind these.

There are several key tips to keep in mind for model quality assurance.

1. Firstly, ensure the box is centered tightly around the desired object with minimal background. This prevents the AI from learning unnecessary background information and ensures better model accuracy.

2. Be sure that the image quality is up to standard and can be easily identified by you first. If you can't clearly identify the object, don't expect the AI to do so either.

3. Be clear and consistent with your label definitions. For example, two similar pictures below should be using the same label as "little blocked." Ambiguous or confusing labels can result in poor performance, especially when working with a limited number of samples.

Step 6: Move some Training data to Test set

Test set is used to test your model's accuracy after training. EI recommends an approximate 80/20 train/test split ratio for your data for every class (or label) in your dataset. I am being lazy and want to try with less.

So I only move 19 to Testing Data, 4 Clean, 7 little blocked and 8 blocked.

Step 7: Create impulse

At the beginning, I set the camera resolution to 160x160, but now I need to set it back to 96x96. This will resize the samples back to 96x96, allowing the model to be trained with that resolution. This will result in a smaller model and my goal is to achieve fast inferencing speed on A1101.

Check the Output features in the screenshot below, here it means blocked drain is labeled as object 0, clean drain as object 1, and little blocked as object 2. The output from the AI will be in the form of probabilities for each object, expressed as percentages. For example, the output may indicate that there is an 85% chance that the image is object 0 (blocked drain), a 10% chance that it is object 1 (clean drain), and a 5% chance that it is object 2 (little blocked).

Input name and then click Save Impulse.

Step 8 Generate image features

Set the color depth to Grayscale.

Step 9 Check the quality of the picture samples

Click "Generate features" to quickly assess the quality of your pictures. The extracted features are displayed on a map, where better separation between dots indicates clearer features.

In my dataset, the blue dot (blocked) is clearly distinguishable, but the green (little blocked) and orange dots (clean) overlap, which may cause the model to occasionally confuse clean and little blocked images. I double-checked the images behind these two dots and verified that they were correctly classified. However, I understand why the features are not as obvious for these 2 images, which is why they overlap.

If you notice data in the explorer that is difficult to separate, the neural network may also struggle with it.

If you need a complete explanation of this tool, you can visit EI doc.

Step 10 Start Training

To be honest, I'm still not sure what the best training settings are, as it seems to be a process of trial and error. However, I'd like to share my experience so far.

Number of training cycles refers to how many times you feed your datasets to the model. I believe that once the model accuracy hits a peak, additional training cycles won't improve it any further.

Learning rate: an explanation that I copy from chatGPT. It is a key factor in determining how quickly the model learns. If the learning rate is too high, the model may miss the best solution, while a low learning rate may take too long to find a good solution.

Validation set size determines how many pictures are marked out for validation. The default setting is 20%, but in my case, with only 100+ pictures, I found 10% to be sufficient.

Data augmentation is a technique that can improve accuracy, but I'm not very familiar with it. Nonetheless, I decided to take it.

I choose to use FOMO to train my datasets since the model size is small.

First time training result – not good enough with only 75%

After many rounds of optimization – seems good now with 97.3% score.

Step 11 Model testing

Click Model testing.

The Test data consists of pictures that have not been seen by the trained model. These pictures are used to evaluate the performance of the model and ensure that it can generalize well to new, unseen data. By testing the model on this data, we can assess its accuracy and make any necessary adjustments before deploying it.

Step 12 Deployment

Scroll down, choose A1101 and pick quantized optimization since I want faster inferencing on device. You will be able to download a firmware when the model is built.

Step 13: Drag the downloaded firmware to SENSECAP

Make sure A1101 is connected to your laptop. Double click the boot button to enter into the boot mode. Then you will see a new storage drive called SENSECAP. Drag the firmware to SENSECAP.

Step 14 Unplug the type-C cable. Next, we need to configure the A1101 to connect to the cloud.

This enables me to check the inferencing result over the phone.

1. Download SenseCAP Mate APP

2. Choose global server and sign in.

3. Follow the steps below to configure the A1101.

4. Settings

Platform: SenseCAP for TTN.

Algorithm: object detection

Frequency Plan: US915

Uplink interval: I choose 60 mins, means that I will receive a pushed result every 1 hr.

Click Send.

Step 15 check the performance of the model

Here comes the most excited part!!!!! I post different pictures with different level of blockage on my screen and show it to A1101.

Blocked drain was labeled as object 0, clean drain as object 1, and little blocked as object 2. When I showed A1101 a clean drain picture, it accurately predicted that it was 97% object 1 (clean).

Step 16 Scan the QR code on device to add the A1101 in my account

Then I can check the drain blockage level on SenseCAP Mate.

Step 17 Placed A1101 on a plant to guard the drain

Quick summary -- It took me about 8 hours to complete this demo. I spent 4 hours on sampling and labeling, and encountered some challenges with poor picture quality that required retakes. Initially, I thought this would be an easy scenario for AI and only took 60 pictures. However, the model performance was poor, and I had to increase the number of pictures to 196 to achieve better accuracy. As a beginner with no experience in model optimization, it was time-consuming to experiment with the training settings, and it took me another 4 hours to find the best configuration. Now I have more experience with the process, should be faster next time.

I want to express my gratitude to all the people who have contributed to open AI tools and datasets. Thanks to their efforts, people like me without deep programming knowledge can build their own AI sensors.

This project is made public, feel free to clone the project here.

Super fun experience!

Useful links:

Seeed wiki

Clone this project