Banana Ripening State Classifier

Super Speedy Summary (Abstract)

The goal of the project is to develop a mobile application that is able to identify the ripeness stage of a banana using the usual sensors given on a current smartphone.

The Problem

Bananas are awesome, but one has to choose the right bananas to buy for one's use case. Do you plan to eat the bananas all at once? Buy the yellow ones. Do you want to eat them tomorrow or the day(s) after? Maybe prefer green ones. You want to use them for cooking? Choose the mushy brown ones, if you really prefer them. No judgement.

For most people, detecting the right ripening level is trivial. Others, e.g. people with colour vision deficiencies, can have problems to identify ripeness of fruit in general, depending on their type of colour vision alteration - to them, most bananas can appear to be the same colour, which makes things quite difficult.

The Idea

To solve that issue, we wanted to use a smartphone with built-in camera to detect a banana and its current stage of ripeness. The app should be small and convenient - just point your phones camera on the fruit and it should tell you how ripe it is.

Initially, we wanted to use computer vision to detect the banana as a banana itself, via object detection - then distinguish its ripeness by its colour and texture. This more or less complicated (or elaborate) way was scrapped in favour of another approach: training a Neural Network to identify three ripeness stages of bananas given on pictures: "not ripe yet", "ripe" and "overripe".

Intro

This project is created in context of the lecture “IoT Applications Prototyping” at the Brunel University / TAE Esslingen. It covers the invention of an own IoT Idea and Application from the scratch solving a real-world problem. In the lecture the MSc students learn about signal processing, pattern recognition and scientific analysis in context of IoT applications.

MrDio's Smart-Sensing-Framework

... is an application framework for Android smartphones connecting the phones built-in sensors like accelerometers, gyroscopes, microphones, GPS etc.; additionally it involves technologies like OpenCV for computer vision and TensorFlow for machine learning, creating a package to rapidly prototype experimental context aware applications.

IoT Prototyping Framework

The IoT Prototyping Framework (IoTPF) is a collection of tools, modules and samples with the aim to empower students and developers creating full-stack IoT prototypes in a short time period.

Contact:

Idea and Application: Fabian, Julian, Bimba, Alexander, Michael

Lecture, SSF and IoTPF: dionysios.satikidis@gmail.com

Tools:

https://github.com/MrDio/Smartphone-Sensing-Framework

https://github.com/MrDio/IoT-Prototyping-Framework

Brunel University London / TAE Esslingen 2019

Training a Neural Network

To solve the classification problem of recognising a distinct ripening state of a banana, a Neural Network was trained with different sets of images. These image sets where manually categorised in the three ripening process stages "unripe", "ripe" and "overripe", which the Neural Network should later be able to recognise.

The machine learning library TensorFlow was then used to create a "banana ripening state classifier". Given a subset of the three sets of banana images before as the training portion of the labelled data, the Neural Network was trained to identify certain characteristics of the different kinds of bananas. The remaining labelled data was then used to check the training results - validating and testing the accuracy of image classification.

Also, some photoshopped pictures of bananas where created to further test the trained classifier, as the colour information in the images turned out to be the most impactful parameter on the classification.

The classifier recognised the yellow banana to be ripe with over 99% probability, and the artificially green one with almost 96% probability to be unripe.

Our real world testing afterwards usually delivered probabilities of 90% or more while accurately analysing the correct ripening state. Especially the colour is a crucial factor in the detection, which is also affected by the ambient light conditions - in cooler lighting environments the tendencies to detect yellow, ripe bananas as green, unripe bananas definitely increased, which still is a problem to be solved.

APP / GUI

The application shows a live feed from the smartphone camera. In set intervals, the Neural Network analyses a snapshot picture of the camera feed. The analysis result is shown in percentages and with a bar, to clearly visualise the bananas current state.

Application Demonstration

The application in action - it works! :)