SPHARM - A Smart Farm Solution

Hello guys!! Today I am going to talk about a project which can be used as a smart farm solution. I had submitted this project in a competition called Techfest which was organized by IIT Bombay. First of all, I would like to thank my friend Aarush Roy. He helped me in making and submitting this project.

Let me present some stats to you .

• Many crops and plants, all around the world, die due to a lack of moisturized soil.
• Animals can also get into the farm and damage the crops on a large scale.

Temperature increases may cause yield declines between 2.5% and 10% across several agronomic species throughout the 21st century (Hatfield et al., 2011). These were some reasons why we should be careful about crop yields.

I knew about TinyML and I thought it could be the correct place to use the technology.

Locust and Mosquito detection

To do this I have used the TinyML technology. I have used Edge Impulse for this. It took me about 4 days to build my dataset.

It has four categories:

• Noise
• Unknown
• Mosquito
• Grasshopper(also called locust)

Then I used MFE processing block for non voice data and then I trained it with Classification learning block and then I saved it.

I trained the model with 30 epochs and got an accuracy of 90.7 percent and a loss value of 0.43. The confusion matrix stats are also good as there is a minor 10% confusion in mosquito and noise.

Now I had to test the model. I tested it and the results were satisfactory.

The grasshopper count is 5 and it is quite good. You may say that the noise count is greater. That happened because of inadequate data and also because the system tried to look at that instant moment only. A system that calculates the result on the basis of averages can be used here.

The mosquito sample worked perfectly giving us 16 counts out of 19.

Temperature, Humidity and Proximity sensors

I am going to use in built Temperature, Humidity and Proximity sensors on Arduino Nano BLE 33 Sense.

Soil Moisture Sensor

The connections are simple: Vcc - 3v ; Gnd - Gnd; Out - A0. I found it convenient to run on 5v. For doing that, I used Arduino Uno's 5V port.

We will determine soil conditions using "if/else" statements. Also, make sure to calibrate your soil moisture sensor.

Bluetooth features

After classifying and collecting the data it needs to be sent to the farmer's smartphone. For this we will use BLE technology. I have designed an app using Thunkable which can connect to the Arduino using Bluetooth.

Thunkable link - https://x.thunkable.com

The Arduino has some BLE characteristics which send all the data to the connected device. Once you connect your device with BLE 33 Sense, then the app will process the data and show it on the screen.

All devices are shown in the list. My device name was SmartFarm_DT.

The first report is related to mosquito and locust detection. The second, third, fourth reports are for temperature, proximity, soil moisture sensors respectively.

Some other screenshots of the app are given above.

Code

/* Edge Impulse Arduino examples
 * Copyright (c) 2021 EdgeImpulse Inc.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 * SOFTWARE.
 */

// If your target is limited in memory remove this macro to save 10K RAM
#define EIDSP_QUANTIZE_FILTERBANK   0

/**
 * Define the number of slices per model window. E.g. a model window of 1000 ms
 * with slices per model window set to 4. Results in a slice size of 250 ms.
 * For more info: https://docs.edgeimpulse.com/docs/continuous-audio-sampling
 */
#define EI_CLASSIFIER_SLICES_PER_MODEL_WINDOW 3

/* Includes ---------------------------------------------------------------- */
#include <PDM.h>
#include <SmartFarm_inferencing.h>
#include <ArduinoBLE.h>
#include <Arduino_APDS9960.h>
#include <Arduino_HTS221.h>

BLEService gadgetControl("69AA173B-A714-4BD1-9188-E84979999559");

String Proximity_Sensor_Status;
String Data;
String Final;
int SoilMoisture;
String TempVal;
String SoilMoisStat;

BLEStringCharacteristic NotifyCharacteristic("79F10571-EBA1-4B5E-91AD-52BD34E06995", BLERead | BLENotify, 220);
BLEStringCharacteristic TempCharacteristic("6C9F6563-A89A-49A9-BA54-CFD6C19F8F5E", BLERead | BLENotify, 220);
BLEStringCharacteristic ProxCharacteristic("D14850F8-DDC6-4D43-A47F-DDFA71495743", BLERead | BLENotify, 220);
BLEStringCharacteristic SoilCharacteristic("B934996E-4417-44B9-A577-811E6D5C8E93", BLERead | BLENotify, 220);

/** Audio buffers, pointers and selectors */
typedef struct {
    signed short *buffers[2];
    unsigned char buf_select;
    unsigned char buf_ready;
    unsigned int buf_count;
    unsigned int n_samples;
} inference_t;

static inference_t inference;
static bool record_ready = false;
static signed short *sampleBuffer;
static bool debug_nn = false; // Set this to true to see e.g. features generated from the raw signal
static int print_results = -(EI_CLASSIFIER_SLICES_PER_MODEL_WINDOW);

/**
 * @brief      Arduino setup function
 */
void setup()
{
    // put your setup code here, to run once:
    Serial.begin(115200);
    Serial.println("Edge Impulse Inferencing Demo");
    
    //BLE Malfunction check
    if(!BLE.begin())
    {
     Serial.println("Starting BLE Failed!");
     while(1);
    }
    
    //Humidity and temperature sensor malfunction check
    if (!HTS.begin()) 
    {
      Serial.println("Failed to initialize humidity temperature sensor!");
      while (1);
    }
  
     //APDS9960 Malfunction check
    if (!APDS.begin())
    {
      Serial.println("Error initializing APDS9960 sensor!");
    }
    //BLE address vars
    String addres = BLE.address();
    Serial.println("Our BLE address is: "+addres);

    //Name inits and charcteristic intialization
    BLE.setDeviceName("SmartFarm_DT");
    BLE.setLocalName("SmartFarm_DT");
    BLE.setAdvertisedService(gadgetControl);   // add the service to advertise which we made in the beginning 
    gadgetControl.addCharacteristic(NotifyCharacteristic);
    gadgetControl.addCharacteristic(TempCharacteristic);
    gadgetControl.addCharacteristic(ProxCharacteristic);
    gadgetControl.addCharacteristic(SoilCharacteristic);
    BLE.addService(gadgetControl);
    BLE.advertise();
  
    // summary of inferencing settings (from model_metadata.h)
    ei_printf("Inferencing settings:\n");
    ei_printf("\tInterval: %.2f ms.\n", (float)EI_CLASSIFIER_INTERVAL_MS);
    ei_printf("\tFrame size: %d\n", EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE);
    ei_printf("\tSample length: %d ms.\n", EI_CLASSIFIER_RAW_SAMPLE_COUNT / 16);
    ei_printf("\tNo. of classes: %d\n", sizeof(ei_classifier_inferencing_categories) /
                                            sizeof(ei_classifier_inferencing_categories[0]));

    run_classifier_init();
    if (microphone_inference_start(EI_CLASSIFIER_SLICE_SIZE) == false) {
        ei_printf("ERR: Failed to setup audio sampling\r\n");
        return;
    }
}

/**
 * @brief      Arduino main function. Runs the inferencing loop.
 */
void loop()
{
    bool m = microphone_inference_record();
    if (!m) {
        ei_printf("ERR: Failed to record audio...\n");
        return;
    }

    signal_t signal;
    signal.total_length = EI_CLASSIFIER_SLICE_SIZE;
    signal.get_data = &microphone_audio_signal_get_data;
    ei_impulse_result_t result = {0};

    EI_IMPULSE_ERROR r = run_classifier_continuous(&signal, &result, debug_nn);
    if (r != EI_IMPULSE_OK) {
        ei_printf("ERR: Failed to run classifier (%d)\n", r);
        return;
    }
    
    BLEDevice central = BLE.central();
    if (central)
    {
      //Prox sensor reading
      if (APDS.proximityAvailable()) 
      {
        int proximity = APDS.readProximity();
        if(proximity<150){
          Proximity_Sensor_Status =  "Object detected";
          Serial.println(Proximity_Sensor_Status);
      }
      else
      {
        Proximity_Sensor_Status = "Clear from proximity sensor";
      }
     }
     
     //Soil Moisture reading
     SoilMoisture = map(analogRead(A0),440,1023,100,0);
     Serial.println(SoilMoisture);

     //Temp and Hum sensor reading
     float Temp = HTS.readTemperature();
     float Hum = HTS.readHumidity();  
       
     int data1 = result.classification[0].value*100;
     int data2 = result.classification[1].value*100;
     int data3 = result.classification[2].value*100;
       //String st1 = "Chances that it is a mosquito: ";
       //String st2 = "Chances that it is just a noise: "; 
       //String st3 = "Chances that it is a grasshopper: ";      
       //String Final = st1+data1+st2+data2+st3+data3;
     if (data2 >= 50)
     {
       Final = "Mosquitos are most probably breeding. You may make use of an insecticide.";
       NotifyCharacteristic.writeValue(Final); 
       if (Temp>40){
        TempVal = "Temperature too high";                   
       }else{
        TempVal = "Normal Temperature"; 
       }
       if(SoilMoisture >= 45){
        SoilMoisStat="Soil Moisture levels are good!!";
       }else{
        SoilMoisStat="Soil Moisture levels are less!! Needs Water";
       }
       TempCharacteristic.writeValue(TempVal);
       ProxCharacteristic.writeValue("Proximity sensor status: "+ Proximity_Sensor_Status);
       SoilCharacteristic.writeValue(SoilMoisStat);         
     }else if(data1 >= 50)
     {
       Final = "Locusts are most probably breeding. You may make use of an insecticide.";
       NotifyCharacteristic.writeValue(Final);    
       if (Temp>40){
        TempVal = "Temperature too high";                   
       }else{
        TempVal = "Normal Temperature"; 
       }
       if(SoilMoisture >= 45){
        SoilMoisStat="Soil Moisture levels are good!!";
       }else{
        SoilMoisStat="Soil Moisture levels are less!! Needs Water";
       }
       TempCharacteristic.writeValue(TempVal);
       ProxCharacteristic.writeValue("Proximity sensor status: "+ Proximity_Sensor_Status);
       SoilCharacteristic.writeValue(SoilMoisStat);  
     }else if(data3 >= 50)
     {
       Final = "Everything is fine :)";
       NotifyCharacteristic.writeValue(Final);    
       if (Temp>40){
        TempVal = "Temperature too high";                   
       }else{
        TempVal = "Normal Temperature"; 
       }
       if(SoilMoisture >= 45){
        SoilMoisStat="Soil Moisture levels are good!!";
       }else{
        SoilMoisStat="Soil Moisture levels are less!! Needs Water";
       }
       TempCharacteristic.writeValue(TempVal);
       ProxCharacteristic.writeValue("Proximity sensor status: "+ Proximity_Sensor_Status);
       SoilCharacteristic.writeValue(SoilMoisStat);   
     }
       for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
            ei_printf("    %s: %.5f\n", result.classification[ix].label,
                      result.classification[ix].value);}
                      
 }

    if (++print_results >= (EI_CLASSIFIER_SLICES_PER_MODEL_WINDOW)) {
        // print the predictions
        ei_printf("Predictions ");
        ei_printf("(DSP: %d ms., Classification: %d ms., Anomaly: %d ms.)",
            result.timing.dsp, result.timing.classification, result.timing.anomaly);
        ei_printf(": \n");
        for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
            ei_printf("    %s: %.5f\n", result.classification[ix].label,
                      result.classification[ix].value);
        }
#if EI_CLASSIFIER_HAS_ANOMALY == 1
        ei_printf("    anomaly score: %.3f\n", result.anomaly);
#endif

        print_results = 0;
    }
}

/**
 * @brief      Printf function uses vsnprintf and output using Arduino Serial
 *
 * @param[in]  format     Variable argument list
 */
void ei_printf(const char *format, ...) {
    static char print_buf[1024] = { 0 };

    va_list args;
    va_start(args, format);
    int r = vsnprintf(print_buf, sizeof(print_buf), format, args);
    va_end(args);

    if (r > 0) {
        Serial.write(print_buf);
    }
}

/**
 * @brief      PDM buffer full callback
 *             Get data and call audio thread callback
 */
static void pdm_data_ready_inference_callback(void)
{
    int bytesAvailable = PDM.available();

    // read into the sample buffer
    int bytesRead = PDM.read((char *)&sampleBuffer[0], bytesAvailable);

    if (record_ready == true) {
        for (int i = 0; i<bytesRead>> 1; i++) {
            inference.buffers[inference.buf_select][inference.buf_count++] = sampleBuffer[i];

            if (inference.buf_count >= inference.n_samples) {
                inference.buf_select ^= 1;
                inference.buf_count = 0;
                inference.buf_ready = 1;
            }
        }
    }
}

/**
 * @brief      Init inferencing struct and setup/start PDM
 *
 * @param[in]  n_samples  The n samples
 *
 * @return     { description_of_the_return_value }
 */
static bool microphone_inference_start(uint32_t n_samples)
{
    inference.buffers[0] = (signed short *)malloc(n_samples * sizeof(signed short));

    if (inference.buffers[0] == NULL) {
        return false;
    }

    inference.buffers[1] = (signed short *)malloc(n_samples * sizeof(signed short));

    if (inference.buffers[1] == NULL) {
        free(inference.buffers[0]);
        return false;
    }

    sampleBuffer = (signed short *)malloc((n_samples >> 1) * sizeof(signed short));

    if (sampleBuffer == NULL) {
        free(inference.buffers[0]);
        free(inference.buffers[1]);
        return false;
    }

    inference.buf_select = 0;
    inference.buf_count = 0;
    inference.n_samples = n_samples;
    inference.buf_ready = 0;

    // configure the data receive callback
    PDM.onReceive(&pdm_data_ready_inference_callback);

    PDM.setBufferSize((n_samples >> 1) * sizeof(int16_t));

    // initialize PDM with:
    // - one channel (mono mode)
    // - a 16 kHz sample rate
    if (!PDM.begin(1, EI_CLASSIFIER_FREQUENCY)) {
        ei_printf("Failed to start PDM!");
    }

    // set the gain, defaults to 20
    PDM.setGain(127);

    record_ready = true;

    return true;
}

/**
 * @brief      Wait on new data
 *
 * @return     True when finished
 */
static bool microphone_inference_record(void)
{
    bool ret = true;

    if (inference.buf_ready == 1) {
        ei_printf(
            "Error sample buffer overrun. Decrease the number of slices per model window "
            "(EI_CLASSIFIER_SLICES_PER_MODEL_WINDOW)\n");
        ret = false;
    }

    while (inference.buf_ready == 0) {
        delay(1);
    }

    inference.buf_ready = 0;

    return ret;
}

/**
 * Get raw audio signal data
 */
static int microphone_audio_signal_get_data(size_t offset, size_t length, float *out_ptr)
{
    numpy::int16_to_float(&inference.buffers[inference.buf_select ^ 1][offset], out_ptr, length);

    return 0;
}

/**
 * @brief      Stop PDM and release buffers
 */
static void microphone_inference_end(void)
{
    PDM.end();
    free(inference.buffers[0]);
    free(inference.buffers[1]);
    free(sampleBuffer);
}

#if !defined(EI_CLASSIFIER_SENSOR) || EI_CLASSIFIER_SENSOR != EI_CLASSIFIER_SENSOR_MICROPHONE
#error "Invalid model for current sensor."
#endif

I am sorry for the unorganized code. You can rename variables and make sure to customise the UUID and characteristic names. Moreover, the soil moisture part is based on my calibration.

Original Project Layout

My original idea was like this:

But I decided to choose Arduino nano ble 33 sense as I had it available. I will try to make the new system like the one in the image. The system in the image aims to provide cost-efficient services to the farmer.

Some problems I faced

It would be better if I could use ML for classifying temperature and soil moisture data. But I dont think that it is possible to run more than one impulse on an arduino nano ble 33. Is there any solution to this??

Also, the code is unorganized and complex to understand. I am sorry for that.

Comment below.

Conclusion

I can conclude that the project is not ready for practical implementation. I have just made a prototype for the idea.

If you have any suggestions or comments, then please tell me in the comment box below.

Thank You !!