Plastic Bottle Detector For Lake

1.0 PROBLEM STATEMENT

Many people thought plastic bottle/containers are something that will be "digested" (degradable) by our mother nature and just discard them to river/drain/lake/ocean at their convenience. My project is inspired by the famous carton character WALL-E, which is a robot designed to clean up the trash on Earth. It's a good movie that highlight how serious it can be if human being continue dumping trash to mother nature, one day we will need WALL-E to clean up our mess.

2.0 PROPOSAL & SCOPE

The scope of my project is of course not to build a WALL-E, but to create the sensing part of it, as in using the vision to detect plastic bottle afloat on lake and sending the data to the cloud. If time allows, the system can be improved with mechanism to retrieve/remove the detected plastic bottle afloat on the lake.

The project is developed using the SenseCAP K1100 kit provided by Seeed Studio, with only 3 parts to be used:

• Wio Terminal the controller with tft display
• Grove-Vision AI, to run inference for object detection
• Grove-LoRa E5, to send data by connecting to Helium

3.0 HOW TO

The project is divided to 3 main parts:

• Object detection
• Object ID & tracking
• Connectivity

3.1 Object Detection

I'm thrilled to see this Grove-Vision AI can actually run object detection, a feature not common for microcontroller (well, we are seeing some breakthrough in recent years, a popular example will be the FOMO developed by Edge Impulse that run on OpenMV H7 plus). This Grove-Vision AI has the same Himax HX6537-A microcontroller as Himax WiseEye WE-I Plus, which has 32-bit DSP running at 400MHz designed to support TensorFlow Lite deployment. There are some really useful tutorial provided by Seeed Studio to start with, I list down at below for others reference.

• https://wiki.seeedstudio.com/K1100-Vision-AI-Module-Grove-LoRa-E5/
• https://wiki.seeedstudio.com/Train-Deploy-AI-Model-A1101-Grove-Vision-AI/

In my case, the OV2640 camera for my kit is somewhat strange, it's having some color distortion (probably something wrong with the filter) and intermittent connection issue, so I replace with another OV2640 from my other ESP32-Cam board. It looks so much better.

Since my goal is to detect plastic bottle, I need dataset of plastic bottle for convolutional neural network (CNN) training, the link above contains a python script to collect images from the Grove-Vision AI, and annotate the images in Roboflow, an extremely useful web-based app for computer vision AI application. There is a nice lake at the park near me, so a stroll in the park brining my laptop and this little Grove-Vision AI board has been regular exercise for me in the past few weeks.

Free plan in Roboflow can only generate about 2.1k dataset which is sufficient for CNN training to begin with. Follow the steps in the link above, I need to generate the dataset in PyTorch format and I'll train the model using pre-trained YOLOv5 model in Google Colab (the notebook is also included in the link above). The model summary as below, both precision & recall are reasonably good and the mean Average Precision (mAP) at 0.5 IoU is also good, great! The last step of the notebook is to convert the TensorFlow Lite model file to uf2 file, which I need to flash it to the Grove-Vision AI board later.

I was struggling to get the Grove-Vision AI to run inference at first, when I only flash the custom-dataset model uf2 file to the board. The tutorial somehow miss-out the step to tell user to flash another uf2 file first (which I believe is the firmware for configuring the OV2640 camera & Himax MCU, probably OpenCV library plus some bounding box information. The config uf2 file is not open so I wouldn't know what is inside.) before flashing the model uf2 file. Strange.. I can't locate the firmware folder anymore in the tutorial/repo (at the time of writing, 11th Sep 2022), I attach the screenshot of the firmware files that I downloaded in August 2022.

Finally, I can run inference with the Grove-Vision AI, the object count and confidence (of first 4 objects) are displayed in the tft of Wio terminal. Besides, it also display the inference time, you can see that it's quite long, barely 1.5fps. The OV2640 camera is also far from ideal to be deployed to outdoor since the FOV is really limited, and the inference speed is pretty much restricted by MCU (Food for thought: What if we train and deploy monochrome image instead of RGB format? That should be at least twice/triple faster.). Anyway, video below showcase what it's capable as of now.

3.2 Object ID & Tracking (future work)

Technically, the detected object(s) in single frame doesn't translate to useful information, because we can be sure the same the object appears in next frame is the same object in previous frame. The argument is, if I pass on the count of plastic bottles to cloud, I can only only tell that we are seeing this amount of plastic bottles floating on the lake, at that point of time.

Hence the idea of object ID & tracking, each object needs to be labelled with unique ID, for counting over a period of time. This is a computing-expensive feature usually found on computer vision application. I decided to take this part as future work since the project is closing in on the contest deadline, and main reason is the Grove-Vision AI board is not entirely open yet, I don't see how I can play around with that board even though I can retrieve the bounding box x, y, w & h in the Wio through I2C. But that is not efficient because to label unique ID, typical approach is to compute the object distance between frames and determine if it's within the boundary of potential movement. Brief explanation on the steps:

• Compute the centroid x & y of each object in current frame
• Compute the centroid x & y of each object in next frame
• Compute the Euclidean distance between object of current & next frame, the closer the distance the higher probability they are the same object. This is one of the process that really stress the MCU, since it involves dividing & square root operation. Not to forget, there might be multiple objects detected.

That's why I think this is not an easy task for now and I will work on this in future.

3.3 Connectivity

The Helium connection is fairly poor in the area I live, even though the coverage map says there are plenty, which then I realize most of the hotspot are inactive. So, I have to drive around the town and search for connectable hotspot. To test that, I had made a simple display in the Wio Terminal to show the uplink status, and a simple icon that switch color when connected. Video below demo about the light sensor value being sent to Google sheet via Helium. I didn't manage to integrate this to the plastic bottle detection application because the lake that I intend to deploy is not within coverage.

I'd tried to publish the data to the web app shared by LiMengDu in this link. Although I managed to get the dashboard up in my browser (in my local machine), the data didn't get through Azure Hub and make it to the dashboard. I didn't pursue that at the end because the investigation had drained up quite a lot of my Helium DC since I only got the free 250 DC.

I'm trying to figure out why the my Grove-LoRa-E5 can't connect to the hostpot even though the coverage map shows that the hotpot is active. I suspect could it be the brand of gateway manufacturer, when I check at the map where I have successful connection, the hotspot are SenseCAP, Linxdot, Nebra; whereas those active hotspot that refuse my connection is Heltec, which unfortunately I'm surrounded by all Heltec hotspot. I have thought to invest one gateway at some point but was held back by the hefty price tag ($519 for SenseCAP M1 AS923)

4.0 CONCLUSION

As explained above, the connectivity are very limited in the country that I live in, perhaps the adoption of LoRa is still very early stage at here or there is no/low interest in my country. Anyway, I decided to submit the project as it is since I can't connect the plastic bottle detector (the Wio Terminal + Grove-Vision AI) to Helium at the lake that I intend to deploy this application. Nonetheless, this is a good experience for me, as I'm being introduced to the world of LoRa (& Helium), and hopefully LoRa will be more widely accepted here in the near future. Plus, I hope the Grove-Vision-AI can be open source so that the community can contribute to idea/project for this useful device.