Plastic waste detection to reduce river pollution

PROBLEM STATEMENT

Challenge#3. Many people still thinks plastic materials are something that will be degradable by our mother nature and just discard them to river/drain/lake/ocean at their convenience. The presence of plastic bottle/containers not only endangering wildlife but also clogging the river/drain that cause flash flood, which occurs quite often in my country.

PROPOSAL & SCOPE

The scope of my project is a Vision system with edge computing and connectivity to identify plastic bottle/container floating on river/drain. This system to serve as first level defender to water pollution. If time allows, the system can be improved with mechanism to extract/remove the detected plastic bottle/container from river/drain.

The project is developed using the SenseCAP K1100 kit provided by Seeed Studio, with only 3 parts to be used:

• Wio Terminal the controller with tft display
• Grove-Vision AI, to run inference for object detection
• Grove-LoRa E5, to send data by connecting to Helium

The project is divided to 3 main parts:

• Object detection
• Object ID & tracking
• Connectivity

Object Detection

I'm thrilled to see this Grove-Vision AI can actually run object detection, a feature not common for microcontroller (well, we are seeing some breakthrough in recent years, a popular example will be the FOMO developed by Edge Impulse that run on OpenMV H7 plus). This Grove-Vision AI has the same Himax HX6537-A microcontroller as Himax WiseEye WE-I Plus, which has 32-bit DSP running at 400MH designed to support TensorFlow Lite deployment. There are some really useful tutorial provided by Seeed Studio to start with, I list down at below for others reference.

• https://wiki.seeedstudio.com/K1100-Vision-AI-Module-Grove-LoRa-E5/
• https://wiki.seeedstudio.com/Train-Deploy-AI-Model-A1101-Grove-Vision-AI/

In my case, the OV2640 camera for my kit is somewhat strange, it's having some color distortion (probably something wrong with the filter) and intermittent connection issue, so I replace with another OV2640 from my other ESP32-Cam board. It looks so much better.

Since my goal is to detect plastic bottle, I need dataset of plastic bottle for convolutional neural network (CNN) training, the link above contains a python script to collect images from the Grove-Vision AI, and annotate the images in Roboflow, an extremely useful web-based app for computer vision AI application. There is a nice lake at the park near me, so a stroll in the park brining my laptop and this little Grove-Vision AI board has been regular exercise for me in the past few weeks.

Free plan in Roboflow can only generate about 2.1k dataset which is sufficient for CNN training to begin with. Follow the steps in the link above, I need to generate the dataset in PyTorch format and I'll train the model using pre-trained YOLOv5 model in Google Colab (the notebook is also included in the link above). The model summary as below, both precision & recall are reasonably good and the mean Average Precision (mAP) at 0.5 IoU is also good, great! The last step of the notebook is to convert the TensorFlow Lite model file to uf2 file, which I need to flash it to the Grove-Vision AI board later.

I was struggling to get the Grove-Vision AI to run inference at first, when I only flash the custom-dataset model uf2 file to the board. The tutorial somehow miss-out the step to tell user to flash another uf2 file first (which I believe is the firmware for configuring the OV2640 camera & Himax MCU, probably OpenCV library plus some bounding box information. The config uf2 file is not open so I wouldn't know what is inside.) before flashing the model uf2 file. Strange.. I can't locate the firmware folder anymore in the tutorial/repo (at the time of writing, 11th Sep 2022), I attach the screenshot of the firmware files that I downloaded in August 2022.

Finally, I can run inference with the Grove-Vision AI, the object count and confidence (of first 4 objects) are displayed in the tft of Wio terminal. Besides, it also display the inference time, you can see that it's quite long, barely 1.5fps. The OV2640 camera is also far from ideal to be deployed to outdoor since the FOV is really limited, and the inference speed is pretty much restricted by MCU (Food for thought: What if we train and deploy monochrome image instead of RGB format? That should be at least twice/triple faster.). Anyway, video below showcase what it's capable as of now.

Object ID & Tracking (future work)

Technically, the detected object(s) in single frame doesn't translate to useful information, because we can be sure the same the object appears in next frame is the same object in previous frame. The argument is, if I pass on the count of plastic bottles to cloud, I can only only tell that we are seeing this amount of plastic bottles floating on the lake.

Hence the idea of object ID & tracking, each object needs to be labelled with unique ID, for counting over a period of time. This is a computing-expensive feature usually found on computer vision application. I decided to take this part as future work since the project is closing in on the contest deadline, and main reason is the Grove-Vision AI board is not entirely open yet, I don't see how I can play around with that board even though I can retrieve the bounding box x, y, w & h in the Wio through I2C. But that is not efficient because to label unique ID, typical approach is to compute the object distance between frames and determine if it's within the boundary of potential movement. Brief explanation on the steps:

• Compute the centroid x & y of each object in current frame
• Compute the centroid x & y of each object in next frame
• Compute the Euclidean distance between object of current & next frame, the closer the distance the higher probability they are the same object. This is one of the process that really stress the MCU, since it involves dividing & square root operation. Not to forget, there might be multiple objects detected.

That's why I think this is not an easy task for now and I will work on this in future.

Connectivity

The Helium connection is fairly poor in the area I live, even though the coverage map says there are plenty, which then I realize most of the hotspot are inactive. So, I have to drive around the town and search for connectable hotspot. To test that, I had made a simple display in the Wio Terminal to show the uplink status, and a simple icon that switch color when connected. Video below demo about the light sensor value being sent to Google sheet via Helium. I didn't manage to integrate this to the plastic bottle detection application because the lake that I intend to deploy is not within coverage.

I'd tried to publish the data to the web app in this link. Although I managed to get the dashboard up in my browser, the data didn't get through Azure Hub and make it to the dashboard.

OUTCOME

As explained above, the hotspots are very limited in the country that I live in, perhaps the adoption of LoRa is still very early stage at here or there is no/low interest in my country. Anyway, I decided to submit the project as it is since I can't connect the plastic bottle detector (the Wio Terminal + Grove-Vision AI) to Helium at the lake that I intend to deploy this application. Nonetheless, this is a good experience for me, as I'm being introduced to the world of LoRa (& Helium), and hopefully LoRa will be more widely accepted here in the near future.