MoldSense

A while back, I realized the bread we were buying sometimes expired too early, or expired way past the expiration date. I realized the expiration date on a loaf of bread wasn't really a clear indicator of whether it was expired or not, and wanted to build something that could help find out what stage a slice of bread was at.

To begin with, I researched the indicators that would most likely show mold activity:

• temperature inside of the bread bag
• humidity inside of the bread bag
• air pressure
• gas volatility
• and the pH of the actual bread

These indicators were based off of two previous research studies I found:

IoT Based Detection of Molded Bread and Expiry Prediction using Machine Learning Techniques - EUDL

Early detection of spoilage moulds in bread using volatile production patterns and quantitative enzyme assays | Journal of Applied Microbiology | Oxford Academic (oup.com)

The first one combines a CO2 sensor with humidity/precipitation, while the second only uses gas sensors. This told me that gas sensors and humidity were good indicators, since they had been used reliably before. While searching for sensors for humidity and gas I found the BME688, which included temperature and pressure as well, and was relatively low cost.

The thing I noticed missing from both studies, however, was the usage of pH sensors. pH is usually a reliable indicator of biological activity, so I wanted to try and use it for this project.

BME688

For the first four measures (temp., humidity, air pressure, gas volatility), I found a very nice and easy to use board: the Adafruit BME688 sensor

This sensor combined all 4 initial measurements into one. I played around with this board for a while to make sure it was working. However, after just a few days, the sensor stopped responding properly- it wasn't changing any values and was stuck at ridiculously high values. This took a week off my build time as I had to debug the board.

I realized I had fried the board while soldering on header pins improperly. After getting a new board, I made sure it was calibrated properly and that it was working fine. After a few more tests, I finally moved on to my last measure: The pH of the bread.

Bread pH

This was fairly tricky since I realized pH sensors were really only used to measure the pH of liquids. After a while searching for different options, I found the Sper Scientific spear tip pH probe. This probe was designed differently- it used a spear tip instead of a glass ball to measure the pH. This meant that the probe could first insert itself into the bread then use the moisture inside to measure the bread's pH.

The Raspberry Pi

I realized although I was going to participate in an AMD contest and was going to use AMD hardware, I needed something small and simple to connect all these sensors together. For this purpose, I used the Raspberry Pi. I already owned a Raspberry Pi and it was simple enough to connect to all of the sensors and spin up a web server for something else to connect to.

I then encountered another problem with my setup, however- the pH sensor emitted an analog signal, but the Raspberry Pi only accepted digital signals. I used Adafruit's ADS1115 to solve this problem. The ADS1115 takes the signal from the pH sensor and converts it into a digital signal the Pi can use.

As shown in the uploaded diagram from the attachments tab, I then connected all of my sensors together:

The left sensor is the BME688, the top is the ADS1115, and the right/bottom is the pH sensor, all of them coming back into the Raspberry Pi.

Training

Now since I was going to be using a model of some sorts, I needed to collect training data. To do so, I took a few slices of bread and cut them into 4 pieces each to get more surfaces for training. I then put these pieces into ziploc bags- 2 pieces per bag. I slid the BME688 inside and held the pH probe to the slice of bread for about half a minute per piece. While doing so, I ran a script I wrote to collect data and save it as a csv file. (The collectdata.py file in the uploaded code)

The results of this csv can be found as pre_training.csv and post_training.csv, for data from the bread before and after it visually started growing mold (respectively). This step took a few weeks as I needed to wait until the bread visually started growing mold to confirm it had actually expired.

Using the data

I initially trained and used a model using the Naive Bayes Classifier- this work can be seen in the "train.py" file. However, I realized this was providing really low accuracy, and was not using the Ryzen AI NPU. To use the Ryzen AI NPU, I switched to a version of Llama 3 instead. I chose the Llama 3 because it was the most powerful LLM I found that could be run on the NPU.

Setting up the NPU

To begin, I found this helpful guide for setting up the NPU - Running LLM on AMD NPU Hardware - Hackster.io

After I was set up, I tried finding models to run on the NPU. I initially only found image processing and object detection models, which weren't quite helpful since that would only help me detect whether there was mold on my bread or not. If mold could be seen with a camera, there wasn't really a need for a program to tell you your bread was expired. The entire goal of my project was to use hidden measures to guesstimate whether the mold had already started growing or not.

However, as mentioned before, I then found a version of the Llama 3 LLM I could run on the NPU: dahara1/llama3-8b-amd-npu · Hugging Face

I set this up and was quickly able to get started.

However, I soon realized my training data was far too hard to use- the data collection script produced thousands of rows which I couldn't easily feed into an LLM. I first converted all of the data into a time series average, since all of the data was really just a collection of time series.

The code used to convert the data into an average time series can be found in find_average.py

Finally, a quick test with Llama 3 showed that the data was working fine:

I fed the time series average for pre_training and post_training to Llama 3, then data for a slice of bread that had been left out for a while. Llama 3 looked at both datasets and outputted "74.92%" as its estimate, which was fairly close.

The last step to the project was an attempt to make this more consumer friendly. I made a small webpage where users could start analyzing their bread, and the webpage would interface with the Llama 3 model to estimate how expired their bread was. This code is the server.py and templates folder.

Continuation Notes

As I started late (I required Ryzen AI for the LLM) and this was my first time with many things- for example my first time interfacing with these kinds of sensors as well as my first time playing around with NPUs and AI, I quickly ran out of time. However, there are plenty of other things I plan to do for this project in the future-

• An enclosure can be designed and made so that the sensor wires are hidden and the project is more consumer friendly
• Llama 3 can be fine tuned a little more specifically for data analysis
• Add a microscopic camera to allow for visual inspection of mold spores