DeViridi: IoT Food Spoilage Sensor and Monitoring Dashboard

Overview: Food Spoilage and Harvest Loss

The Rockefeller Foundation estimates 470 million smallholder farmers and supply chain actors across developing countries lose an average of 15 per cent of their income to food spoilage. It is estimated a solution for food spoilage would feed one billion more people by 2050. The main issues farmers in developing countries across the world face is food storage and detection of spoiled foods. This comes at a cost to both farmers and processing companies. Food waste (FLW) across the agriculture sector is an increasingly growing contributor to greenhouse emissions, with an estimated 50% of post-harvest crops wasted in Kenya (Global, 2015) . This has unparalleled effects on the environment with each wasted product not only incurring the loss of edible food but contributing to methane production, which United States Environmental Protection Agency estimated to be 25 times more efficient at trapping heat than carbon dioxide. Food waste occurs due to pests and storage conditions, with many farmers, pack houses and exporters throwing away yield due to appearance and fears of spoilage. Current methods of determining spoilage remain qualitative, but the detection of released gases and computer vision provides a more effective solution. A low-cost solution is required which not only improves food preservation but can also detect food spoilage to assure food quality.

Potential for Domestic Usage: Homes or Supermarkets

A key source of food waste in developed countries is the disposal of food is due to the influence of ‘best-before’ and other expiration dates. There are currently three date label phrases commonly used on foods across the world. The first is “Sell By” which informs the retailer the last date the product can be sold and is not indicative of safety or quality. The second is “Best Before” which propositions the date the food’s quality may decline. The final is the “Use By” which recommends to consumers the final date the product should be consumed (Tsiros and Heilman, 2018). These labels are usually not endorsed by and association (i.e., United States Department of Agriculture) and are simply guidelines, which do not take into account the storing conditions of the product. Research shows consumers often disposed of products which were past the stated date of expiration without any further evaluation (i.e., visual cues). A study conducted in Oregon wanted to determine the reasons for food wastage (Van Garde and Woodburn, 1987). With a sample of 243 households, personal interviews, and 7-day discard records were collected along with discards over 3 days from a subsample of 50. The results showed the aesthetic factors were the main reason for food disposal for the 18-25-year-old group, but for the above 65 group, the major reason was personal experience with food storage. The study concluded consumers did not have clear criteria in the safety evaluation of foods. Many studies have shown age is a factor in the interpretation of labels (Van Boxstael et al., 2014).

Based on experimental data, more quantitative methods for detecting food waste could be implemented to reduce food waste from supermarkets and households.

Identifying Rotting

When foods rot, they can release gases based on their composition. An example is the production of ammonia (NH3), hydrogen sulphide (H2S) in the rotting of meats. By detecting the composition of the air around meat we can determine the extent of rotting and the subsequent edibility of the food. This can provide a more accurate prediction on the end-life of a food product. Fruits and vegetables can also produce ethylene (C2H4) to indicate freshness (Saltveit, 1999). Decaying food can produce methane- a potent greenhouse gas. This can sometimes be detected in many fruits, vegetables and meat but is non-specific for the majority of rotting foods (Smith et al., 2009). Moreover, the use of AI-based image detection could be used to detect patterns in food spoilage.

A combination of both image and gas measures could provide a more accurate method to identify spoilage.

Main Deliverables

A proof of concept device to accurately detect food spoilage and rotting for use in :

i) Farms and Post-Harvest Suppliers

ii) Supermarket and Households

Section 1 Computer Vision Model: Food Spoilage

Step 1: Obtaining Image Dataset

The most important aspect of creating a good predictive model involves the dataset being used. Roboflow has their own public datasets available, which are handy to use in this project. You can search for them here: https://universe.roboflow.com

For the proof of concept we will use a dataset created by Brac University. Brac University. ( 2022 ). Fruit Freshness Detection Dataset

https://universe.roboflow.com/brac-university-v9w2y/fruit-freshness-detection-08shj

Click on the download dataset button, select the Yolo V5 model and copy the code.

Want to use your own dataset of images ? I recommend following this tutorial here: https://wiki.seeedstudio.com/Train-Deploy-AI-Model/ Once again it makes use of roboflow, but alternatives such as Edge Impulse will be easier to use.

Step 2: Training Computer Vision Model

Navigateto : https://colab.research.google.com/gist/lakshanthad/b47a1d1a9b4fac43449948524de7d374/yolov5-training-for-sensecap-a1101.ipynb#scrollTo=Fg4hIoyzE-Qe

Here you will find a colab book, where we will be able to train the model and export a file to our sensor.

Follow all of the steps on the colab notebook and you can begin training the model. Based on your hardware, training can take between minutes to hours ! After training make to sure to download the trained .uf2 model/

Step 3: Porting Trained Model onto Hardware

Find the grove camera sensor and plug into the computer.

Double-click the boot button- you should now be able to see a storage drive on the computer. Drag and drop the model onto the sensor. You have now successfully ported the model onto the camera !

Section 2: Building with Wio Terminal

Step 4: Connecting Sensors to the Wio Terminal

First attach the camera and lora module using the connectors provided in the kit. The camera should be attached to the left slot.

The next aspect to understand is the pinout in the back of the terminal.

Fortunately, the pinout follows that of the RPI, so sensor layouts should be similar to that with the RPI.

Attach the VOC/CO2 sensor:

Sensor to Wio

VCC to 5V (2)

GND to GND (6)

SDA to SDA (3)

SCL to SCL (5)

Step 5: UX Design with Figma

Before we start working with the UX in the Arduino IDE, we should make a simple mockup to help visualise the GUI. Figma is an easy tool to use, ensuring the pixel size of the WIO is adhered to (320 x 240).

Bear in mind the left and top values, as these are were the WIO terminal will place any elements.

Step 6: Coding the GUI on Arduino IDE

The wiki page: https://wiki.seeedstudio.com/Wio-Terminal-LCD-Graphics/ has information on the different functions involved in the WIO Terminal display. Firstly, install the TFT_eSPI.h library (see GitHub for library list) and add it to the IDE (SKETCH--> Include Library --> Add.Zip Library). The below code is some pseudocode to create a similar GUI as above (actual code in github)

#include <TFT_eSPI.h>

TFT_eSPI tft; //Initializing TFT LCD
TFT_eSprite spr = TFT_eSprite(&tft); //Initializing buffer
#define TFT_OLIVE 0xB674



void setup() {
  Serial.begin(9600);
  pinMode(WIO_LIGHT, INPUT);
  tft.begin();
  tft.setRotation(3); //Set TFT LCD rotation
  spr.createSprite(TFT_HEIGHT,TFT_WIDTH); //Create buffer

}
void loop() {
  int lightSensorValue = analogRead(WIO_LIGHT);
  spr.fillSprite(TFT_WHITE);
  spr.fillRect(0,0,320,30,TFT_OLIVE);
  spr.setTextColor(TFT_WHITE);
  spr.setTextSize(3);
  spr.drawString("Food Sensor",68,6);
   spr.drawFastVLine(150,30,210,TFT_DARKGREEN); //Drawing verticle line
  spr.drawFastHLine(0,140,320,TFT_DARKGREEN); //Drawing horizontal line

  //Setting temperature
  spr.setTextColor(TFT_BLACK);
  spr.setTextSize(2);
  spr.drawString("Temperature",10,45);
  spr.setTextSize(3);

   spr.setTextSize(2);
  spr.drawString("Humidity",25,160);
  spr.setTextSize(3);


   spr.setTextSize(2);
  spr.drawString("Gas Concent",160,45);
  spr.setTextSize(3);

  spr.setTextSize(2);
  spr.drawString("Light",200,160);
  spr.setTextSize(3);
  lightSensorValue = map(lightSensorValue,0,1023,0,100); //Map sensor values 
  spr.drawNumber(lightSensorValue,205,190); //Display sensor values as percentage  
  spr.drawString("%",245,190);
  spr.pushSprite(0,0);
  delay(50);
  
}

For the display to show up and update, you need to push the created sprite. All the colours use the RGB565 system and you can use this tool to generate your own colours: http://www.barth-dev.de/online/rgb565-color-picker/

Step 7: Pre-Requisite Sensor Libraries

Check the list on GitHub to install all the different libraries for the sensors and connectivity !

Step 8: Using State Control and Programming Sensor Values

The program uses state control, which is helpful for debugging. The order of operations are:

1) Get Temperature and Humidity

2) Get Gas Sensor Value

3) Get Camera Classification

4) (IF NEEDED) Trigger Warning for Rotten Food

5) Send Data to SD Card and Lora

REPEAT

You can download the code from the GitHub.

Food Rotting Algorithm

The SGP30 sensor can measure an TVOC signal between 0 to 60,000 ppb based on the datasheet https://www.mouser.com/pdfdocs/Sensirion_Gas_Sensors_SGP30_Datasheet_EN-1148053.pdf

Studies have shown when bananas begin to over-ripe and rot, they can release both ethylene and C02 in varying concentrations (Ramírez-Sánchez et al., 2018) . Since ethylene is a TVOC, the SGP 30 sensor is perfectly suited to detect both ethylene and CO2.

Near the end of the cycle, the rates tend to 1.3 ngkg-1s-1 which is 1.2 ppb per second. Hence, considering the production of both gases during an interval of time and calculating the equivalent rate of gas production would be used e.g. 5 seconds would cause a change in 6ppb. For reliability, the alarm will only be triggered if the daily average of the individual scores is bigger than the threshold.

This will be used in conjunction with the visual database. A confidence of 90% is recommended, with the average confidence rate of images across the day taken.

Step 9: Connecting to LoRa and Helium Network

Add the custom decoder code from the GitHub and here: https://wiki.seeedstudio.com/Connecting-to-Helium/ for any other sensors

Connect the function and device using the flows panel.

Step 10: Enclosure Design and 3D Printing

3D print the attached mini-enclosure and fit the device into the slots. Wire management is included.

Final Product

Videos

Videos available here:

https://drive.google.com/drive/folders/1-KCYGheQV1h4DoS2HanhG7r7WA9LgLHB?usp=sharing

Sources

Van Boxstael, S. et al. (2014) ‘Understanding and attitude regarding the shelf life labels and dates on pre-packed food products by Belgian consumers’, Food Control, 37(1), pp. 85–92. doi: 10.1016/J.FOODCONT.2013.08.043.

European Commission (2018) Date marking and food waste | Food Safety, European Commission . Available at: https://ec.europa.eu/food/safety/food_waste/eu_actions/date_marking_en (Accessed: 1 August 2022).

European Commission and TSN Political and Social (2015) Flash Eurobarometer 425: Food waste and date marking, Directorate-General for Communication. Available at: http://data.europa.eu/euodp/en/data/dataset/S2095_425_ENG (Accessed: 1 August 2022).

Van Garde, S. J. and Woodburn, M. J. (1987) ‘Food discard practices of householders’, Journal of the American Dietetic Association, 87(3), pp. 322–329.

Jarvis, A. and Gardiner, J. (2018) Market study on date marking and other information provided on food labels and food waste prevention. doi: 10.2875/808514.

Global, F. (2015) Food waste in Kenya: Uncovering food waste in the horticultural export supply chain. Available at: https://feedbackglobal.org/wp-content/uploads/2015/07/Food-Waste-in-Kenya_report-by-Feedback.pdf.

De Laurentiis, V., Caldeira, C. and Sala, S. (2020) ‘No time to waste: assessing the performance of food waste prevention actions’, Resources, Conservation and Recycling, 161, p. 104946. doi: 10.1016/J.RESCONREC.2020.104946.

Ramírez-Sánchez, M. et al. (2018) ‘Physiological, molecular and ultrastructural analyses during ripening and over-ripening of banana (Musa spp., AAA group, Cavendish sub-group) fruit suggest characteristics of programmed cell death’, Journal of the Science of Food and Agriculture, 98(2), pp. 609–617. doi: 10.1002/JSFA.8505.

Sawa, D. B. (2019) The truth about expired food: how best-before dates create a waste mountain, The Guardian [online]. Available at: https://www.theguardian.com/food/2019/apr/17/the-truth-about-expired-food-how-best-before-dates-create-a-waste-mountain (Accessed: 1 August 2022).

Tsiros, M. and Heilman, C. M. (2018) ‘The Effect of Expiration Dates and Perceived Risk on Purchasing Behavior in Grocery Store Perishable Categories’:, https://doi.org/10.1509/jmkg.69.2.114.60762, 69(2), pp. 114–129. doi: 10.1509/JMKG.69.2.114.60762.

WRAP (2016) Quantification of food surplus , waste and related materials in the grocery supply chain. Available at: https://wrap.org.uk/resources/report/quantification-food-surplus-waste-and-related-materials-supply-chain (Accessed: 1 August 2022).