Preparing the DL Hub

Published December 8, 2020 © MIT

Interacting with the Flora

Improvising on Precision Farming technique, through combined efforts of DL & sensing Electrophysiological response from plant.

AdvancedFull instructions providedOver 6 days1,064

Things used in this project

Hardware components

Avnet Ultra96-V2

DFRobot FireBeetle ESP32 IOT Microcontroller (Supports Wi-Fi & Bluetooth)

Adafruit Waterproof DS18B20 Digital temperature sensor

Rohm BH1750 Light Intensity Sensor

SparkFun Single Lead Heart Rate Monitor - AD8232

Digilent 60W PCIe 12V 5A Power Supply

USB-A to Micro-USB Cable

Thin wire electrodes and conductive jelly.

Software apps and online services

AMD PYNQ Framework

Xilinx Vitis-AI

Jupyter Notebook

Arduino IDE

Hand tools and fabrication machines

Soldering Station, Hobbyist

Mimosa Pudica 5-month sapling

Story

In 1873, following correspondence with Charles Darwin, electrical signals in plants were discovered by Burdon-Sanderson during a survey of Dionaea muscipula(Venus Flytrap). Based on this work, Jagadish Chandra Bose, at the beginning of the twentieth century, showed that plants generated electrical impulses in response to environmental stimuli similar to those of nerves in animals.

In plants, there are different types of electrical signals involving changes in membrane potentials that could encode electrical information related to physiological states when plants are stimulated by different environmental conditions. These functions may include growth, gas exchange, respiration, variation of photosynthesis and transpiration, and modification of gene expression(e.g. protease inhibitor).Through automatic classiﬁcation one can identify environmental cues that can act as plant-stressors and utilizing machine-learning techniques that can further be applied for biotic stress identification in precision plant growth and protection.

Dive-in through Mimosa Pudica (SHY_GUY)

In a healthy Mimosa plant, you can observe two "rapid movement" responses to touch. With a light touch brushed along the leaves (called pinnules), the leaves fold together at points (pulvinules) along the rib (rachis). With a strong touch, the leaves will fold and the branch will drop along the point (pulvinus) where the main branch (petiole) joins the stem. That further initiates an electrical impulse (Action Potential) propagation along the rachis that results in the plant movement.

Preparing the EndNode for sensing Floral Stimulus.

A Proof of Concept device has been tailored utilizing Low Power BLE capabilities of ESP32, AD8232 instrumentation Op-Amp, BH1750 light intensity sensor and waterproof DS18B20 digital temperature sensor for measuring the apparent plant and soil characteristic. Each end-node can be powered over battery, and proves to be a cost-effective model while scaling things up. Imagine the possibilities of forecasting plant diseases and vector, Phyto-actuating mechanisms like irrigation-valves and artificial light-control, all performed solely by plant/crop itself.

MiMo_Sense communicates over UART as well as BLE

A, BH1750 lux sensor is connected over I2C (D21 & D22) and programmed for continuous measurement, while DS18B20 is connected to (D15) pulled-up with 2.7K in reference to Vsup(3.3V)

MiMo_Sense getting ready to spike things 'UP'

DS18B20 probe with ground ref electrode

Output from AD8232 is fed to VP (A0) of ESP32, while the input is connected to thin wire electrodes with ground reference electrode inserted to the root region.

(A) Reference courtesy Backyard Brains (both A & B)

(B) A dab of conductive jelly! (No hot glue this time)

Processingthe SIGS__|\__/\__/|__

Each of the plant electrical response was taken at a difference of 3 hours, for getting response at pinnule and pulvinus point. Moreover, the response from DS18b20 embedded within soil, remained at a constant value of 22 degree Celsius (probably the soil was too soggy). The lux sensor showed decent drift in light intensity during the entire course of experiment and was found to be 300lx during the dusk hours.

The 'Light-Touch' and 'Strong-Touch' recovery time were found to be approximately 22 minutes and 40 minutes respectively.

(A) Pinnules response at rachis [Light-Touch]

(B) Pulvinus response at rachis[Strong-Touch]

Preparing the DL Hub

Pre-paration scripts for Ultra96V2 (Pynq2.5)

During the course of the project, I had faced connectivity issues regarding onboard BLE and WiFi access on Ultra96V2, thanks to Xilinx and Element14 community for providing necessary guidance and one-shot scripts based on Pynq 2.5 architecture. I myself would like to share a few.

- While connecting with USB gadget Ethernet support, there persists connectivity issues for connecting Ultra96V2 to internet over WiFi AP, particularly due to hardware design and or, OS specific support. The wifi.ipynb script makes success while accessing the Ultra96V2 over Jtag2Serial Pod, but misses badly over USB-Ethernet gadget, since no wlan is shown in ifconfig. Workaround:

sudo python3

>>> from pynq.lib import Wifi
>>> port = Wifi()
>>> port.connect("SSID","PASSWORD")

- Activating Bluetooth service on Ultra96V2:

echo BT_POWER_UP > /dev/wilc_bt
echo BT_DOWNLOAD_FW > /dev/wilc_bt
stty -F /dev/ttyPS1 115200

hciattach /dev/ttyPS1 -t 10 any 115200 noflow nosleep
sleep 1s
hciconfig hci0 up
hciconfig hci0

Ensure you can use Bluetooth services like bluetoothctl like any other Linux distribution.

Updating the PYNQ with Vitis-AI feature:

- Open new terminal window on homepage and enter each one in succession.

git clone --recursive --shallow-submodules https://github.com/Xilinx/DPU-PYNQ
cd DPU-PYNQ/upgrade
make

- Install Python DPU Package:

pip3 install pynq-dpu

- Run & Build from available docker script

cd Vitis-AI
./docker_run.sh xilinx/vitis-ai:latest

cd ./docker
./docker_build_cpu.sh

Alongside, do install matplotlib, pyserial, grabserial (for sensor dataset acquisition), bluepy, keras, tensorflow.

Readying with the Communication

- Wired Method: Fetching sensor data from End-node through grabserial in Ultra96V2 terminal can be done by relaying the following command; (have a checkin for lsusb)

grabserial -v -d "/dev/ttyUSB0" -b 115200 -w 8 -p N -s 1 -e 90 -t -m "Starting kernel*"

MiMo_Sense packet acquisition through serial.

ensuring connectivity through bluetoothctl

Running 'Sensor Data acquisition over BLE for Ultra96V2.ipynb' script, streams sensor data through Bluetooth low energy link onto Ultra96V2.

Dataset was manually labelled, since each of the category only contained fewer samples to proceed with the ML model.

Category A (Osmotic Root Pressure), was obtained as an arbitrary value with soil temperature coefficient. This category consisted of 80 signals, that were obtained after post-processing.
Category B (Ambient Temperature), was obtained as per exact values obtained from DS18b20 sensor pushed out from the soil, and 40 signals were collected in this case (non-stress state).
Category C (Light Intensity), produced over 150 signals during the measurement phase. Out of which approx. 60 were collected at end during hard-touch and stressed-state.

Interval Arithmetic algorithm is utilized to learn features extracted from plant-biosignal. After feature extraction, it is further piped into SVM, in order to classify whether Mimosa(target plant) has been subjected to environmental stress. SVM classifier is utilized through LibSVM library available for python.

Conclusion

The next goal will be to ﬁnd an algorithm that can be successfully applied to learn a pattern, even when trained on smaller datasets, as in our case. Various algorithms have been provided by H2O.ai, an “Open Source Fast Scalable Machine Learning Platform”, that could be utilized for improving on the original concept. Ultra96V2 combined with Pynq altogether provides powerful tool to those that would love to implement ML and DL applications of their own. I have high hopes to continue with Pynq 2.6 in my next project.

Thanks Hackster and Xilinx.

Code

// Sensing environmental response[Temperature, Lux, BiopotentialVal] in Mimosa Pudica and relaying over BLE through ESP32::Ultra96V2.
#include <OneWire.h>
#include <DallasTemperature.h>
#define ONE_WIRE_BUS 15
OneWire oneWire(ONE_WIRE_BUS);
DallasTemperature sensors(&oneWire);

#include <Wire.h>
#include <BH1750.h>

#include <BLEDevice.h>
#include <BLEServer.h>
#include <BLEUtils.h>
#include <BLE2902.h>

BLEServer* pServer = NULL;
BLECharacteristic* pCharacteristic = NULL;
bool deviceConnected = false;
bool oldDeviceConnected = false;
BH1750 lightMeter;

#define SERVICE_UUID "6E400001-B5A3-F393-E0A9-E50E24DCCA9E"  //Nordic UART Service
#define CHARACTERISTIC_UUID "6E400003-B5A3-F393-E0A9-E50E24DCCA9E"

class MyServerCallbacks: public BLEServerCallbacks {
    void onConnect(BLEServer* pServer) {
      deviceConnected = true;
    };

    void onDisconnect(BLEServer* pServer) {
      deviceConnected = false;
    }
};

int BPV; 
boolean BPVval = false;
int sensorPin = A0;        //pin number to use the ADC
int sensorValue = 0;      //initialization of sensor variable, equivalent to EMA Y
 
float EMA_a_low = 0.3;    //initialization of EMA alpha
float EMA_a_high = 0.5;
 
int EMA_S_low = 0;        //initialization of EMA S
int EMA_S_high = 0;
 
int highpass = 0;
int bandpass = 0;

void setup() {
  Serial.begin(115200);
  Wire.begin();
  pinMode(12, INPUT);
  pinMode(13, INPUT);
  sensors.begin();
  lightMeter.begin();
  
  EMA_S_low = analogRead(sensorPin);      //set EMA S for t=1
  EMA_S_high = analogRead(sensorPin);
  
  // Create the BLE Device
  BLEDevice::init("MiMo_Sense");

  // Create the BLE Server
  pServer = BLEDevice::createServer();
  pServer->setCallbacks(new MyServerCallbacks());

  // Create the BLE Service
  BLEService *pService = pServer->createService(SERVICE_UUID);

  // Create a BLE Characteristic
  pCharacteristic = pService->createCharacteristic(
                      CHARACTERISTIC_UUID,
                      BLECharacteristic::PROPERTY_READ   |
                      BLECharacteristic::PROPERTY_WRITE  |
                      BLECharacteristic::PROPERTY_NOTIFY |
                      BLECharacteristic::PROPERTY_INDICATE
                    );

  pCharacteristic->addDescriptor(new BLE2902());

  // Start the service
  pService->start();

  // Start advertising
  BLEAdvertising *pAdvertising = BLEDevice::getAdvertising();
  pAdvertising->addServiceUUID(SERVICE_UUID);
  pAdvertising->setScanResponse(false);
  pAdvertising->setMinPreferred(0x0);  // set value to 0x00 to not advertise this parameter
  BLEDevice::startAdvertising();
  Serial.println("Waiting a client connection to notify...");
}

void loop() {
  sensors.requestTemperatures();
  
  int temperature = sensors.getTempCByIndex(0);  
  int lux = lightMeter.readLightLevel();
  if((digitalRead(12) == 1)||(digitalRead(13) == 1)){
    Serial.println("Leads Off!");
    //BPV = 0;
    BPVval = false;
  }
  else{    
      //Serial.print("BPV: ");
      //Serial.println(analogRead(A0));
      //BPV = analogRead(A0);
      BPVval = true;  }
  sensorValue = analogRead(sensorPin);    //read the sensor value using ADC
  
  EMA_S_low = (EMA_a_low*sensorValue) + ((1-EMA_a_low)*EMA_S_low);  //run the EMA
  EMA_S_high = (EMA_a_high*sensorValue) + ((1-EMA_a_high)*EMA_S_high);
   
  //highpass = sensorValue - EMA_S_low;     //find the high-pass as before (for comparison)
  //bandpass = EMA_S_high - EMA_S_low;      //find the band-pass
  
  BPV = EMA_S_high - EMA_S_low;
  // notify changed value
  if (deviceConnected) {
    
     char TempStr [2];
     char BPVStr [2];
     char LuxStr [2];
     dtostrf (temperature, 1, 2, TempStr);
     dtostrf (BPV, 1, 2, BPVStr);
     dtostrf (lux, 1, 2, LuxStr);
     char MiMoDataStr [16];
     sprintf (MiMoDataStr, "% d,% d,% d" , temperature, lux, BPV);
     pCharacteristic->setValue(MiMoDataStr);
     pCharacteristic->notify();
     Serial.println(MiMoDataStr);
     delay(3); // bluetooth stack will go into congestion, if too many packets are sent!
    }
    // disconnecting
    if (!deviceConnected && oldDeviceConnected) {
        delay(500); // give the bluetooth stack the chance to get things ready
        pServer->startAdvertising(); // restart advertising
        Serial.println("start advertising");
        oldDeviceConnected = deviceConnected;
    }
    // connecting
    if (deviceConnected && !oldDeviceConnected) {
        // do stuff here on connecting
        oldDeviceConnected = deviceConnected;
    }
}

# BLE communication script for ULTRA96V2 and MiMo_Sense. 

from bluepy import btle
import time
import binascii

# Address of BLE device to connect to.

BLE_ADDRESS = "30:AE:A4:75:29:72"

# NORDIC UART service.

BLE_SERVICE_UUID ="6e400001-b5a3-f393-e0a9-e50e24dcca9e"

# MiMo_Sense characteristic (with BLE descriptor{2902}), [Temperature, Lux, BiopotentialVal]

BLE_CHARACTERISTIC_UUID= "6e400003-b5a3-f393-e0a9-e50e24dcca9e";


class MyDelegate(btle.DefaultDelegate):

    def __init__(self):

        btle.DefaultDelegate.__init__(self)

        # ... initializations.


    def handleNotification(self, cHandle, data):

    #	data = bytearray(data)
    	data = bytearray(data)

    	print (data)

	print ('Establishing connection with the client')

dev = btle.Peripheral(BLE_ADDRESS)

dev.setDelegate( MyDelegate() )

service_uuid = btle.UUID(BLE_SERVICE_UUID)

ble_service = dev.getServiceByUUID(service_uuid)

uuidConfig = btle.UUID(BLE_CHARACTERISTIC_UUID)

data_chrc = ble_service.getCharacteristics(uuidConfig)[0]

# print "Debug Services..."

# for svc in dev.services:

# 	print str(svc)

# print 'Debug Characteristics...'

# for ch in es_service.getCharacteristics():

# 	print str(ch)

# Enable the sensor, start notifications
# Writing x01 is the protocol for all BLE notifications.
#data_chrc.write(bytes("\x01")) 

time.sleep(3.0) # Allow to stabilise

# Main loop --------

while True:

    if dev.waitForNotifications(1.0):

        # handleNotification() was called

        continue

    print ("Waiting...")