Kutluhan Aktar
Published © CC BY

IoT AI-assisted Deep Algae Bloom Detector w/ Blues Wireless

Take deep algae images w/ a borescope, collect water quality data, train a model, and get informed of the results over WhatsApp via Notecard

ExpertFull instructions provided4,159

Things used in this project

Hardware components

Blues Notecarrier Pi Hat
×1
Blues Notecard (Cellular)
Blues Notecard (Cellular)
×1
Raspberry Pi 4 Model B
Raspberry Pi 4 Model B
×1
3-in-1 Waterproof USB Borescope Camera
×1
Arduino Nano R3
Arduino Nano R3
×1
DFRobot Analog pH Sensor Pro Kit
×1
DFRobot Analog TDS Sensor
×1
Adafruit Waterproof DS18B20 Digital temperature sensor
Adafruit Waterproof DS18B20 Digital temperature sensor
×1
7'' HDMI Display with Capacitive Touchscreen
DFRobot 7'' HDMI Display with Capacitive Touchscreen
×1
Keyes 10mm RGB LED Module (140C05)
×1
Button (6x6)
×2
Creality Sermoon V1 3D Printer
×1
Creality Sonic Pad
×1
Creality CR-200B 3D Printer
×1
4.7K Resistor
×1
Solderless Breadboard Half Size
Solderless Breadboard Half Size
×1
SparkFun Solder-able Breadboard - Mini
SparkFun Solder-able Breadboard - Mini
×1
Jumper wires (generic)
Jumper wires (generic)
×1

Software apps and online services

Edge Impulse Studio
Edge Impulse Studio
Blues Notehub.io
Blues Notehub.io
Twilio API for WhatsApp
Twilio API for WhatsApp
Arduino IDE
Arduino IDE
Thonny
Fusion
Autodesk Fusion
Ultimaker Cura

Hand tools and fabrication machines

Hot glue gun (generic)
Hot glue gun (generic)

Story

Read more

Custom parts and enclosures

iot_deep_algae_detector_main_case.stl

iot_deep_algae_detector_front_cover.stl

Edge Impulse Model (Linux ARMv7 Application)

Schematics

Notecard

Notecarrier Pi Hat

Code

main.py

Python
# IoT AI-assisted Deep Algae Bloom Detector w/ Blues Wireless
#
# Raspberry Pi 4
#
# Take deep algae images w/ a borescope, collect water quality data,
# train a model, and get informed of the results over WhatsApp via Notecard. 
#
# By Kutluhan Aktar

import cv2
import serial
import json
import notecard
from periphery import I2C
from threading import Thread
from time import sleep
import os
import datetime
from edge_impulse_linux.image import ImageImpulseRunner

class deep_algae_detection():
    def __init__(self, modelfile):
        # Define the required settings to connect to Notehub.io.
        productUID = "<_product_UID_>"
        device_mode = "continuous"
        port = I2C("/dev/i2c-1")
        self.card = notecard.OpenI2C(port, 0, 0)
        sleep(5)
        # Connect to the given Notehub.io project.
        req = {"req": "hub.set"}
        req["product"] = productUID
        req["mode"] = device_mode
        rsp = self.card.Transaction(req)
        print("Notecard Connection Status:")
        print(rsp)
        # Initialize serial communication with Arduino Nano to obtain water quality sensor measurements and the given commands.
        self.arduino_nano = serial.Serial("/dev/ttyUSB0", 115200, timeout=1000)
        # Initialize the borescope camera feed.
        self.camera = cv2.VideoCapture(0)
        # Define the Edge Impulse FOMO model settings.
        dir_path = os.path.dirname(os.path.realpath(__file__))
        self.modelfile = os.path.join(dir_path, modelfile)
        self.detection_results = ""        
        
    def get_transferred_data_packets(self):
        # Obtain the transferred sensor measurements and commands from Arduino Nano via serial communication.
        if self.arduino_nano.in_waiting > 0:
            data = self.arduino_nano.readline().decode("utf-8", "ignore").rstrip()
            if(data.find("Run") >= 0):
                print("\nRunning an inference...")
                self.run_inference()
            if(data.find("Collect") >= 0):
                print("\nCapturing an image... ")
                self.save_img_sample()
            if(data.find("{") >= 0):
                self.sensors = json.loads(data)
                print("\nTemperature: " + self.sensors["Temperature"])
                print("pH: " + self.sensors["pH"])
                print("TDS: " + self.sensors["TDS"])
        sleep(1)

    def run_inference(self):
        # Run an inference with the FOMO model to detect potential deep algae bloom.
        with ImageImpulseRunner(self.modelfile) as runner:
            try:
                # Print the information of the Edge Impulse FOMO model converted to a Linux (ARMv7) application (.eim).
                model_info = runner.init()
                print('Loaded runner for "' + model_info['project']['owner'] + ' / ' + model_info['project']['name'] + '"')
                labels = model_info['model_parameters']['labels']
                # Get the latest frame captured by the borescope camera, resize it depending on the given model, and run an inference.
                model_img = self.latest_frame 
                features, cropped = runner.get_features_from_image(model_img)
                res = runner.classify(features)
                # Obtain the prediction (detection) results for each label (class).
                results = 0
                if "bounding_boxes" in res["result"].keys():
                    print('Found %d bounding boxes (%d ms.)' % (len(res["result"]["bounding_boxes"]), res['timing']['dsp'] + res['timing']['classification']))
                    for bb in res["result"]["bounding_boxes"]:
                        # Count the detected objects:
                        results+=1
                        print('\t%s (%.2f): x=%d y=%d w=%d h=%d' % (bb['label'], bb['value'], bb['x'], bb['y'], bb['width'], bb['height']))
                        # Draw bounding boxes for each detected object on the resized (cropped) image.
                        cropped = cv2.rectangle(cropped, (bb['x'], bb['y']), (bb['x'] + bb['width'], bb['y'] + bb['height']), (0, 255, 255), 1)
                # Save the modified image by appending the current date & time to its file name.
                date = datetime.datetime.now().strftime("%Y-%m-%d_%H_%M_%S")
                filename = 'detections/DET_{}.jpg'.format(date)
                cv2.imwrite(filename, cv2.cvtColor(cropped, cv2.COLOR_RGB2BGR))
                # After running an inference successfully, transfer the detection results
                # and the obtained water quality sensor measurements to Notehub.io.
                if results == 0:
                    self.detection_results = "Algae Bloom  Not Detected!"
                else:
                    self.detection_results = "Potential Algae Bloom  {}".format(results) 
                print("\n" + self.detection_results)
                self.send_data_to_Notehub(self.detection_results)
                
            # Stop the running inference.    
            finally:
                if(runner):
                    runner.stop() 

    def display_camera_feed(self):
        # Display the real-time video stream generated by the borescope camera.
        ret, img = self.camera.read()
        cv2.imshow("Deep Algae Bloom Detector", img)
        # Stop the video stream if requested.
        if cv2.waitKey(1) != -1:
            self.camera.release()
            cv2.destroyAllWindows()
            print("\nCamera Feed Stopped!")
        # Store the latest frame captured by the borescope camera.
        self.latest_frame = img
        
    def save_img_sample(self):    
        date = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
        filename = './samples/IMG_{}.jpg'.format(date)
        # If requested, save the recently captured image (latest frame) as a sample.
        cv2.imwrite(filename, self.latest_frame)
        print("\nSample Saved Successfully: " + filename)
    
    def send_data_to_Notehub(self, results):
        # Send the model detection results and the obtained water quality sensor measurements transferred by Arduino Nano
        # to the webhook (twilio_whatsapp_sender) by performing an HTTP GET request via Notecard through Notehub.io.
        req = {"req": "web.get"}
        req["route"] = "TwilioWhatsApp"
        query = "?results=" + self.detection_results + "&temp=" + self.sensors["Temperature"] + "&pH=" + self.sensors["pH"] + "&TDS=" + self.sensors["TDS"]
        req["name"] = query.replace(" ", "%20")
        rsp = self.card.Transaction(req)
        print("\nNotehub Response:")
        print(rsp)
        sleep(2)
        
        
# Define the algae object.
algae = deep_algae_detection("model/iot-ai-assisted-deep-algae-bloom-detector-linux-armv7-v1.eim")

# Define and initialize threads.
def borescope_camera_feed():
    while True:
        algae.display_camera_feed()
        
def activate_received_commands():
    while True:
        algae.get_transferred_data_packets()

Thread(target=borescope_camera_feed).start()
Thread(target=activate_received_commands).start()

update_web_detection_folder.py

Python
# IoT AI-assisted Deep Algae Bloom Detector w/ Blues Wireless
#
# Raspberry Pi 4
#
# Take deep algae images w/ a borescope, collect water quality data,
# train a model, and get informed of the results over WhatsApp via Notecard. 
#
# By Kutluhan Aktar

import requests
from glob import glob
from time import sleep

# Define the webhook path for transferring the given images to the server  save_img.php.
webhook_img_path = "https://www.theamplituhedron.com/twilio_whatsapp_sender/save_img.php"
# Obtain all image files in the detections folder.
files = glob("./detections/*.jpg")

def send_image(file_path):
    files = {'captured_image': open("./"+file_path, 'rb')}
    # Make an HTTP POST request to the webhook so as to send the given image file.
    request = requests.post(webhook_img_path, files=files)
    print("Image File Transferred: " + file_path)
    # Print the response from the server.
    print("App Response => " + request.text + "\n")
    sleep(2)

# If the detections folder contains images, send each retrieved image file to the webhook via HTTP POST requests
# in order to update the detections folder on the server.
if(files): 
    i = 0
    t = len(files)
    print("Detected Image Files: {}\n".format(t))
    for img in files:
        i+=1
        print("Uploading Images: {} / {}".format(i, t))
        send_image(img)
else:
    print("No Detection Image Detected!")

iot_deep_algae_bloom_detector_sensors.ino

Arduino
       /////////////////////////////////////////////
      //  IoT AI-assisted Deep Algae Bloom       //
     //      Detector w/ Blues Wireless         //
    //             ---------------             //
   //             (Arduino Nano)              //
  //             by Kutluhan Aktar           //
 //                                         //
/////////////////////////////////////////////

//
// Take deep algae images w/ a borescope, collect water quality data, train a model, and get informed of the results over WhatsApp via Notecard.
//
// For more information:
// https://www.theamplituhedron.com/projects/IoT_AI_assisted_Deep_Algae_Bloom_Detector_w_Blues_Wireless
//
//
// Connections
// Arduino Nano :
//                                DFRobot Analog pH Sensor Pro Kit
// A0   --------------------------- Signal
//                                DFRobot Analog TDS Sensor
// A1   --------------------------- Signal
//                                DS18B20 Waterproof Temperature Sensor
// D2   --------------------------- Data
//                                Keyes 10mm RGB LED Module (140C05)
// D3   --------------------------- R
// D5   --------------------------- G
// D6   --------------------------- B
//                                Control Button (R)
// D7   --------------------------- +
//                                Control Button (C)
// D8   --------------------------- +


// Include the required libraries.
#include <OneWire.h>
#include <DallasTemperature.h>

// Define the water quality sensor pins:  
#define pH_sensor   A0
#define tds_sensor  A1

// Define the pH sensor settings:
#define pH_offset 0.21
#define pH_voltage 5
#define pH_voltage_calibration 0.96
#define pH_array_length 40
int pH_array_index = 0, pH_array[pH_array_length];

// Define the TDS sensor settings:
#define tds_voltage 5
#define tds_array_length 30
int tds_array[tds_array_length], tds_array_temp[tds_array_length];
int tds_array_index = -1;

// Define the DS18B20 waterproof temperature sensor settings:
#define ONE_WIRE_BUS 2
OneWire oneWire(ONE_WIRE_BUS);
DallasTemperature DS18B20(&oneWire);

// Define the RGB LED pins:
#define redPin     3
#define greenPin   5
#define bluePin    6

// Define the control button pins:
#define button_r   7
#define button_c   8

// Define the data holders:
float pH_value, pH_r_value, temperature, tds_value;
long timer = 0, r_timer = 0, t_timer = 0;

void setup(){
  // Initialize the hardware serial port (Serial) to communicate with Raspberry Pi via serial communication. 
  Serial.begin(115200);

  // RGB:
  pinMode(redPin, OUTPUT);
  pinMode(greenPin, OUTPUT);
  pinMode(bluePin, OUTPUT);
  adjustColor(0,0,0);

  // Buttons:
  pinMode(button_r, INPUT_PULLUP);
  pinMode(button_c, INPUT_PULLUP);  

  // Initialize the DS18B20 sensor.
  DS18B20.begin();

}

void loop(){   
  if(millis() - timer > 20){
    // Calculate the pH measurement every 20 milliseconds.
    pH_array[pH_array_index++] = analogRead(pH_sensor);
    if(pH_array_index == pH_array_length) pH_array_index = 0;
    float pH_output = avr_arr(pH_array, pH_array_length) * pH_voltage / 1024;
    pH_value = 3.5 * pH_output + pH_offset;

    // Calculate the TDS measurement every 20 milliseconds.
    tds_array[tds_array_index++] = analogRead(tds_sensor);
    if(tds_array_index == tds_array_length) tds_array_index = 0;
    
    // Update the timer.  
    timer = millis();
  }
  
  if(millis() - r_timer > 800){
    // Get the accurate pH measurement every 800 milliseconds.
    pH_r_value = pH_value + pH_voltage_calibration;
    //Serial.print("pH: "); Serial.println(pH_r_value);

    // Obtain the temperature measurement in Celsius.
    DS18B20.requestTemperatures(); 
    temperature = DS18B20.getTempCByIndex(0);
    //Serial.print("Temperature: "); Serial.print(temperature); Serial.println(" C");

    // Get the accurate TDS measurement every 800 milliseconds.
    for(int i=0; i<tds_array_length; i++) tds_array_temp[i] = tds_array[i];
    float tds_average_voltage = getMedianNum(tds_array_temp, tds_array_length) * (float)tds_voltage / 1024.0;
    float compensationCoefficient = 1.0 + 0.02 * (temperature - 25.0);
    float compensatedVoltage = tds_average_voltage / compensationCoefficient;
    tds_value = (133.42*compensatedVoltage*compensatedVoltage*compensatedVoltage - 255.86*compensatedVoltage*compensatedVoltage + 857.39*compensatedVoltage)*0.5;
    //Serial.print("TDS: "); Serial.print(tds_value); Serial.println(" ppm\n\n");

    // Update the timer.
    r_timer = millis();
  }

  if(millis() - t_timer > 3000){
    // Every three seconds, transfer the collected water quality sensor measurements in the JSON format to Raspberry Pi via serial communication. 
    String data = "{\"Temperature\": \"" + String(temperature) + " C\", "
                  + "\"pH\": \"" + String(pH_r_value) + "\", "
                  + "\"TDS\": \"" + String(tds_value) + "  ppm\"}";
    Serial.println(data);

    adjustColor(255,0,255);
    delay(2000);
    adjustColor(0,0,0);
    
    // Update the timer.
    t_timer = millis();
    
  }

  // Send commands to Raspberry Pi via serial communication.
  if(!digitalRead(button_r)){ Serial.println("Run Inference!"); adjustColor(0,255,0); delay(1000); adjustColor(0,0,0); }
  if(!digitalRead(button_c)){ Serial.println("Collect Data!"); adjustColor(0,0,255); delay(1000); adjustColor(0,0,0); }
}

double avr_arr(int* arr, int number){
  int i, max, min;
  double avg;
  long amount=0;
  if(number<=0){ Serial.println("ORP Sensor Error: 0"); return 0; }
  if(number<5){
    for(i=0; i<number; i++){
      amount+=arr[i];
    }
    avg = amount/number;
    return avg;
  }else{
    if(arr[0]<arr[1]){ min = arr[0];max=arr[1]; }
    else{ min = arr[1]; max = arr[0]; }
    for(i=2; i<number; i++){
      if(arr[i]<min){ amount+=min; min=arr[i];}
      else{
        if(arr[i]>max){ amount+=max; max=arr[i]; } 
        else{
          amount+=arr[i];
        }
      }
    }
    avg = (double)amount/(number-2);
  }
  return avg;
}

int getMedianNum(int bArray[], int iFilterLen){  
  int bTab[iFilterLen];
  for (byte i = 0; i<iFilterLen; i++) bTab[i] = bArray[i];
  int i, j, bTemp;
  for (j = 0; j < iFilterLen - 1; j++) {
    for (i = 0; i < iFilterLen - j - 1; i++){
      if (bTab[i] > bTab[i + 1]){
        bTemp = bTab[i];
        bTab[i] = bTab[i + 1];
        bTab[i + 1] = bTemp;
      }
    }
  }
  if ((iFilterLen & 1) > 0) bTemp = bTab[(iFilterLen - 1) / 2];
  else bTemp = (bTab[iFilterLen / 2] + bTab[iFilterLen / 2 - 1]) / 2;
  return bTemp;
}

void adjustColor(int r, int g, int b){
  analogWrite(redPin, (255-r));
  analogWrite(greenPin, (255-g));
  analogWrite(bluePin, (255-b));
}

index.php

PHP
<?php

# Include the Twilio PHP Helper Library.
require_once '../twilio-php-main/src/Twilio/autoload.php'; 
use Twilio\Rest\Client;

# Define the Twilio account information.
$account = array(	
				"sid" => "<_SID_>",
				"auth_token" => "<_AUTH_TOKEN_>",
				"registered_phone" => "+__________",
				"verified_phone" => "+14155238886"
		   );
		
# Define the Twilio client object.
$twilio = new Client($account["sid"], $account["auth_token"]);

# Send a WhatsApp text message from the verified phone to the registered phone.
function send_text_message($twilio, $account, $text){
	$message = $twilio->messages 
              ->create("whatsapp:".$account["registered_phone"],
                        array( 
                            "from" => "whatsapp:".$account["verified_phone"],       
                            "body" => $text							
                        ) 
              );
			  
    echo '{"body": "WhatsApp Text Message Send..."}';
}

# If requested, send a WhatsApp media message from the verified phone to the registered phone.
function send_media_message($twilio, $account, $text, $media){
	$message = $twilio->messages 
              ->create("whatsapp:".$account["registered_phone"],
                        array( 
                            "from" => "whatsapp:".$account["verified_phone"],       
                            "body" => $text,
                            "mediaUrl" => $media						
                        ) 
              );
			  
    echo "WhatsApp Media Message Send...";
}

# Obtain the transferred information from Notecard via Notehub.io.
if(isset($_GET["results"]) && isset($_GET["temp"]) && isset($_GET["pH"]) && isset($_GET["TDS"])){
	$date = date("Y/m/d_h:i:s");
	// Send the received information via WhatsApp to the registered phone so as to notify the user.
	send_text_message($twilio, $account, " $date\n\n"
	                                 ." Model Detection Results:\n "
									 .$_GET["results"]
									 ."\n\n Water Quality:"
									 ."\n Temperature: ".$_GET["temp"]
									 ."\n pH: ". $_GET["pH"]
									 ."\n TDS: ".$_GET["TDS"]
	            );
	
	
}else{
	echo('{"body": "Waiting Data..."}');
}

# Obtain all image files transferred by Raspberry Pi in the detections folder.
function get_latest_detection_images($app){
	# Get all images in the detections folder.
	$images = glob('./detections/*.jpg');
    # Get the total image number.
	$total_detections = count($images);
	# If the detections folder contains images, sort the retrieved image files chronologically by date.
	$img_file_names = "";
	$img_queue = array();
	if(array_key_exists(0, $images)){
		usort($images, function($a, $b) {
			return filemtime($b) - filemtime($a);
		});
		# After sorting image files, save the retrieved image file names as a list and create the image queue by adding the given web application path to the file names.
		for($i=0; $i<$total_detections; $i++){
			$img_file_names .= "\n".$i.") ".basename($images[$i]);
			array_push($img_queue, $app.basename($images[$i]));
		}
	}
	# Return the generated image file data.
	return(array($total_detections, $img_file_names, $img_queue));
}

# If the registered (Twilio-verified) phone transfers a message (command) to this webhook via WhatsApp:
if(isset($_POST['Body'])){
	switch($_POST['Body']){
		case "Latest Detection":
			# Get the total model detection image number, the generated image file name list, and the image queue. 
			list($total_detections, $img_file_names, $img_queue) = get_latest_detection_images("https://www.theamplituhedron.com/twilio_whatsapp_sender/detections/");
			# If the get_latest_detection_images function finds images in the detections folder, send the latest detection image to the verified phone via WhatsApp.
			if($total_detections > 0){
				send_media_message($twilio, $account, " Total Saved Images  ".$total_detections, $img_queue[0]);
			}else{
				# Otherwise, send a notification (text) message to the verified phone via WhatsApp.
				send_text_message($twilio, $account, " Total Saved Images  ".$total_detections."\n\n No Detection Image Found!");
			}
			break;
		case "Oldest Detection":
			# Get the total model detection image number, the generated image file name list, and the image queue.
			list($total_detections, $img_file_names, $img_queue) = get_latest_detection_images("https://www.theamplituhedron.com/twilio_whatsapp_sender/detections/");
			# If the get_latest_detection_images function finds images in the detections folder, send the oldest detection image to the verified phone via WhatsApp.
			if($total_detections > 0){
				send_media_message($twilio, $account, " Total Saved Images  ".$total_detections, $img_queue[$total_detections-1]);
			}else{
				# Otherwise, send a notification (text) message to the verified phone via WhatsApp.
				send_text_message($twilio, $account, " Total Saved Images  ".$total_detections."\n\n No Detection Image Found!");
			}
			break;
		case "Show List":
			# Get the total model detection image number, the generated image file name list, and the image queue.
			list($total_detections, $img_file_names, $img_queue) = get_latest_detection_images("https://www.theamplituhedron.com/twilio_whatsapp_sender/detections/");
			# If the get_latest_detection_images function finds images in the detections folder, send all retrieved image file names as a list to the verified phone via WhatsApp.
			if($total_detections > 0){
				send_text_message($twilio, $account, " Image List:\n".$img_file_names);
			}else{
				# Otherwise, send a notification (text) message to the verified phone via WhatsApp.
				send_text_message($twilio, $account, " Total Saved Images  ".$total_detections."\n\n No Detection Image Found!");
			}
            break;			
		default:
			if(strpos($_POST['Body'], "Display") !== 0){
				send_text_message($twilio, $account, " Wrong command. Please enter one of these commands:\n\nLatest Detection\n\nOldest Detection\n\nShow List\n\nDisplay:<IMG_NUM>");
			}else{
				# Get the total model detection image number, the generated image file name list, and the image queue.
				list($total_detections, $img_file_names, $img_queue) = get_latest_detection_images("https://www.theamplituhedron.com/twilio_whatsapp_sender/detections/");
				# If the requested image exists in the detections folder, send the retrieved image with its file name to the verified phone via WhatsApp.
				$key = explode(":", $_POST['Body'].":none")[1];
				if(array_key_exists($key, $img_queue)){
					send_media_message($twilio, $account, " ". explode("/", $img_queue[$key])[5], $img_queue[$key]);
				}else{
					send_text_message($twilio, $account, " Image Not Found: ".$key.".jpg");
				}
			}
			break;
	}
}

?>

save_img.php

PHP
<?php

// If Raspberry Pi transfers a model detection image to update the server, save the received image to the detections folder.
if(!empty($_FILES["captured_image"]['name'])){
	// Image File:
	$captured_image_properties = array(
	    "name" => $_FILES["captured_image"]["name"],
	    "tmp_name" => $_FILES["captured_image"]["tmp_name"],
		"size" => $_FILES["captured_image"]["size"],
		"extension" => pathinfo($_FILES["captured_image"]["name"], PATHINFO_EXTENSION)
	);
	
    // Check whether the uploaded file extension is in the allowed file formats.
	$allowed_formats = array('jpg', 'png');
	if(!in_array($captured_image_properties["extension"], $allowed_formats)){
		echo 'FILE => File Format Not Allowed!';
	}else{
		// Check whether the uploaded file size exceeds the 5MB data limit.
		if($captured_image_properties["size"] > 5000000){
			echo "FILE => File size cannot exceed 5MB!";
		}else{
			// Save the uploaded file (image).
			move_uploaded_file($captured_image_properties["tmp_name"], "./detections/".$captured_image_properties["name"]);
			echo "FILE => Saved Successfully!";
		}
	}
}

?>

Credits

Kutluhan Aktar

Kutluhan Aktar

81 projects • 307 followers
AI & Full-Stack Developer | @EdgeImpulse | @Particle | Maker | Independent Researcher

Comments

Related channels and tags
  • 3D Printing
  • Adafruit
  • Arduino
  • Autodesk
  • Blues
  • DFRobot
  • Edge Impulse
  • Home Automation
  • Internet of Things
  • Machine Learning & AI
  • Raspberry Pi
  • SparkFun
  • Twilio