The Premise
The Hardware
Setting Up TinyML
Reading Color
The Scanning Rig
Gathering Data
Processing and Sending the Color Information
Training a Model
Testing

Created July 13, 2021 © GPL3+

Determining a Plant's Health with TinyML

Scan the leaves of a plant with an Arduino Nano 33 BLE Sense and train a model to detect if it's diseased.

IntermediateFull instructions provided10 hours808

Grand Prize - US Based Project

2021 China-US Young Maker Competition

Determining a Plant's Health with TinyML

Things used in this project

Hardware components

Arduino Nano 33 BLE Sense

Digilent Stepper Motor

Driver DRV8825 for Stepper Motors for Theremino System

Software apps and online services

Microsoft VS Code

Arduino IDE

Edge Impulse Studio

Hand tools and fabrication machines

3D Printer (generic)

Soldering iron (generic)

Story

The Premise

Just like humans, plants can become diseased too. And just like how you might develop a rash from a skin infection, a plant's leaves might become yellowed and/or blotchy from a fungus or other pathogen. So, by leveraging the power of machine learning, colors can be scanned and then used to train a model that can detect when a leaf's color is off.

The Hardware

The brain of this project is the Arduino Nano 33 BLE Sense, and it was chosen for several reasons. First, it has a rich set of powerful sensors, including a 9DoF IMU, APDS-9960 (color, gesture, proximity, and brightness), microphone, and a temperature/humidity/pressure sensor combo. In order to move the board around the plant's leaf and take measurements, a pair of stepper motors are used in conjunction with a pair of DRV8825 driver boards.

Setting Up TinyML

For this project, the builtin sensors listed for the Arduino Nano 33 BLE Sense on Edge Impulse won't work, since only the accelerator and microphone are listed. This means the data forwarder will have to be used instead of the serial daemon. To begin, I created a new project and named it. Next, I installed the Edge Impulse CLI by installing Node.js and NPM, and then running npm install -g edge-impulse-cli. You might need to add its install path to your PATH environment variable if it can't be found. Next, run edge-impulse-data-forwarder and make sure it works, then use Ctrl+C to exit.

Reading Color

The APDS-9960 reads color by bouncing infrared light off of the surface and reading the wavelengths that aren't absorbed by the material. To communicate with the sensor, it's best to install the Arduino APDS9960 library, which gives access to several useful functions. In the code, the APDS-9960 is first initialized and then the programs enters the loop function. In there, it waits until there is color data. If a reading is available, the color is read with APDS.readColor() along with the proximity to the surface. Each RGB component is converted from a 16-bit number into a ratio of its value over the sum.

The Scanning Rig

Scanning the color of a leaf is accomplished by moving a rig in two axes to pass various locations of the leaf underneath the onboard APDS-9960. Each axis is moved by rotating a lead screw in either a clockwise or counterclockwise motion to translate a block in either direction. The whole system was designed in Fusion 360, and here are some renders of the design below:

The X axis rests on top of the Y axis, letting the top block move in both axes. There is an additional V-wheel on the Y axis to support the weight of the stepper motor. Parts were printed using PLA plastic with around 45% infill.

Gathering Data

When the system first starts up, the stepper motors don't know where they are, so the two axes home by moving to the origin step-by-step until they hit the limit switch. Next, the APDS-9960 is initialized. There is a bounding box that is defined as two two-element arrays that contain opposite corners of a box. A random point is chosen between these two locations, and then the steppers are run to that position while reading the colors in between.

Processing and Sending the Color Information

Colors are read with APDS.readColor(), as previously mentioned. After the sum is calculated, a percentage is calculated and then sent via USB by calling the Serial.printf() method. Values are separated by commas, and each reading is separated by a newline character. When the data is received by the data forwarder program, it is sent to the Edge Impulse cloud project as training data with the given label (as either healthy or unhealthy).

Training a Model

After all of the training data has been collected, it's time to make a model that can differentiate between healthy and unhealthy leaves. I used an impulse comprised of the three-axis time series, a spectral analysis block, and a Keras block. You can see the screenshot below for how I generated the features from the data:

Testing

To test my new model, I gathered some new test data, this time of an unhealthy leaf. The accuracy of the model was around 63%, and after sending through some testing features, it was able to correctly classify the leaf most of the time.

This accuracy can be improved by adding more training data and slowing down the training speed.

Schematics

Code

#include <Arduino_APDS9960.h>
#include <AccelStepper.h>
#include <MultiStepper.h>
#include "pinDefs.h"

int r, g, b, c, p;
float sum;

AccelStepper xStepper(AccelStepper::DRIVER, STEPPER_1_STEP, STEPPER_1_DIR);
AccelStepper yStepper(AccelStepper::DRIVER, STEPPER_2_STEP, STEPPER_2_DIR);

MultiStepper steppers;

// a random location will be chosen within the bounding box
const long boundingBox[2][2] = 
{
    {0, 0},
    {40, 40}
};

void setup()
{
    Serial.begin(115200);
    while(!Serial);

    if(!APDS.begin()) {
        Serial.println("Could not init APDS9960");
        while(1);
    }

    pinMode(X_AXIS_HOMING_SW, INPUT_PULLUP);
    pinMode(Y_AXIS_HOMING_SW, INPUT_PULLUP);
    //Serial.println(digitalRead(X_AXIS_HOMING_SW) + digitalRead(Y_AXIS_HOMING_SW));

    xStepper.setPinsInverted(X_AXIS_DIR);
    yStepper.setPinsInverted(Y_AXIS_DIR);

    xStepper.setMaxSpeed(150);
    yStepper.setMaxSpeed(150);

    steppers.addStepper(xStepper);
    steppers.addStepper(yStepper);
    homeMotors();
}

void loop()
{
    long randomPos[2];
    randomPos[0] = random(boundingBox[0][0], boundingBox[1][0]) * STEPS_PER_MM;
    randomPos[1] = random(boundingBox[0][1], boundingBox[1][1]) * STEPS_PER_MM;

    steppers.moveTo(randomPos);
    while(steppers.run())
    {
        if(!APDS.colorAvailable() || !APDS.proximityAvailable()){}
        else {
            APDS.readColor(r, g, b, c);
            sum = r + g + b;
            p = APDS.readProximity();

            if(!p && c > 10 && sum >= 0)
            {
                float rr = r / sum, gr = g / sum, br = b / sum;
                Serial.printf("%1.3f,%1.3f,%1.3f\n", rr, gr, br);
            }
        }
    }
}

void homeMotors()
{
    // home x
    //Serial.println("Now homing x");
    while(digitalRead(X_AXIS_HOMING_SW))
        xStepper.move(-1);
    // home y
    //Serial.println("Now homing y");
    while(digitalRead(Y_AXIS_HOMING_SW))
        yStepper.move(-1);
    xStepper.setCurrentPosition(0);
    yStepper.setCurrentPosition(0);
}

Credits

Evan Rust

122 projects • 1087 followers

IoT, web, and embedded systems enthusiast. Contact me for product reviews or custom project requests.

Comments

Awards

Grand Prize - US Based Project

2021 China-US Young Maker Competition

Determining a Plant's Health with TinyML