Kutluhan Aktar
Published © CC BY

AI-assisted Pipeline Diagnostics and Inspection w/ mmWave

Extract data items from a mmWave sensor, train a NN to diagnose pipeline defects, and inspect results w/ deformed pipe images on a web app.

ExpertFull instructions provided2 days3,147

Things used in this project

Hardware components

PCBWay Custom PCB
PCBWay Custom PCB
×1
Nicla Vision
Arduino Nicla Vision
×1
Arduino Nano R3
Arduino Nano R3
×1
60GHz mmWave Sensor - Breathing and Heartbeat Module
Seeed Studio 60GHz mmWave Sensor - Breathing and Heartbeat Module
×1
LattePanda 3 Delta
LattePanda 3 Delta
×1
Elecrow 8.8″ (1920*480) IPS Screen
×1
2.8″ 240x320 TFT LCD Touch Screen (ILI9341)
×1
Anycubic Kobra 2
×1
5mm Common Anode RGB LED
×1
SparkFun Logic Level Converter - Bi-Directional
SparkFun Logic Level Converter - Bi-Directional
×3
Button (6x6)
×4
Resistor 220 ohm
Resistor 220 ohm
×3
Power Jack
×1
External Battery
×1
Jumper wires (generic)
Jumper wires (generic)
×1

Software apps and online services

Edge Impulse Studio
Edge Impulse Studio
Arduino IDE
Arduino IDE
KiCad
KiCad
Thonny
Fusion 360
Autodesk Fusion 360
Ultimaker Cura
XAMPP
Visual Studio 2017
Microsoft Visual Studio 2017

Hand tools and fabrication machines

Hot glue gun (generic)
Hot glue gun (generic)
Soldering iron (generic)
Soldering iron (generic)

Story

Read more

Custom parts and enclosures

Gerber Files

Fabrication Files

Edge Impulse Model (Arduino Library)

AI_assisted_Pipeline_Diagnostics_case.stl

AI_assisted_Pipeline_Diagnostics_case_front_cover.stl

AI_assisted_Pipeline_Diagnostics_modular_camera.stl

Elecrow_Module_Left.stl

Elecrow_Module_Right.stl

Elecrow_Module_Cover.stl

Schematics

PCB (1)

PCB (2)

PCB (3)

PCB (4)

PCB (5)

PCB (6)

PCB (7)

Nicla Vision (Pinout)

Code

AI_assisted_Pipeline_Diagnostics_run_model.ino

Arduino
         /////////////////////////////////////////////  
        //    AI-assisted Pipeline Diagnostics     //
       //      and Crack Inspection w/ mmWave     //
      //             ---------------             //
     //          (Arduino Nicla Vision)         //           
    //             by Kutluhan Aktar           // 
   //                                         //
  /////////////////////////////////////////////

//
// Export data items from a 60GHz mmWave sensor, train a NN to diagnose pipeline issues, and inspect model results w/ output images on a web app.
//
// For more information:
// https://www.theamplituhedron.com/projects/AI_assisted_Pipeline_Diagnostics_and_Crack_Inspection_w_mmWave/
//
//
// Connections
// Arduino Nicla Vision :  
//                                Arduino Nano
// UART_TX (PA_9)   --------------- A0
// UART_RX (PA_10)  --------------- A1


// Include the required libraries:
#include <WiFi.h>
#include "camera.h"
#include "gc2145.h"

// Include the Edge Impulse model converted to an Arduino library:
#include <AI-assisted_Pipeline_Diagnostics_inferencing.h>

// Define the required parameters to run an inference with the Edge Impulse model.
#define FREQUENCY_HZ        EI_CLASSIFIER_FREQUENCY
#define INTERVAL_MS         (1000 / (FREQUENCY_HZ + 1))

// Define the features array to classify one frame of data.
float features[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE];
size_t feature_ix = 0;

// Define the threshold value for the model outputs (predictions).
float threshold = 0.60;

// Define the pipeline diagnostic class names:
String classes[] = {"Clogged", "Cracked", "Leakage"};

char ssid[] = "<_SSID_>";        // your network SSID (name)
char pass[] = "<_PASSWORD_>";    // your network password (use for WPA, or use as key for WEP)
int keyIndex = 0;                // your network key Index number (needed only for WEP)

// Define the server on LattePanda 3 Delta.
char server[] = "192.168.1.22";
// Define the web application path.
String application = "/pipeline_diagnostics_interface/update_server.php";

// Initialize the WiFiClient object.
WiFiClient client; /* WiFiSSLClient client; */

// Define the required camera settings for the 2-megapixel CMOS camera (GC2145).
GC2145 galaxyCore;
Camera cam(galaxyCore);

// Define the camera frame buffer.
FrameBuffer fb;

// Create a struct (data) including all 60GHz mmWave sensor data parameters:
struct data {
  float p1;
  float p2;
  float p3;
  float p4;
  float p5;
  float p6;
  float p7;
};

// Define the data holders:
struct data mm;
int predicted_class = -1;
String data_packet = "";
int del_1, del_2, del_3, del_4, del_5, del_6;

void setup(){
  Serial.begin(115200);

  // Initialize the hardware serial port (Serial1) to communicate with Arduino Nano via serial communication. 
  Serial1.begin(115200, SERIAL_8N1);

  // Connect to WPA/WPA2 network. Change this line if using open or WEP network:
  WiFi.begin(ssid, pass);
  // Attempt to connect to the Wi-Fi network:
  while(WiFi.status() != WL_CONNECTED){
    // Wait for the connection:
    delay(500);
    Serial.print(".");
  }
  // If connected to the network successfully:
  Serial.println("Connected to the Wi-Fi network successfully!");

  // Define the pixel format and the FPS settings.
  // Then, initialize the GC2145 camera.
  if (!cam.begin(CAMERA_R320x320, CAMERA_RGB565, 30)) { // CAMERA_R320x240, CAMERA_R320x320
    Serial.println("GC2145 camera: initialization failed!");
  }else{
    Serial.println("GC2145 camera: initialized successfully!");
  }
}

void loop(){
  // Obtain the data packet and commands transferred by Arduino Nano via serial communication.
  if(Serial1.available() > 0){
    data_packet = Serial1.readString();
  }

  if(data_packet != ""){
    
    Serial.print("\nReceived Data Packet: "); Serial.println(data_packet+"\n");
    
    // If Arduino Nano sends the Data command via serial communication:
    if(data_packet.startsWith("Data")){
      // Glean information as substrings from the transferred data packet by Arduino Nano.
      del_1 = data_packet.indexOf("&");
      del_2 = data_packet.indexOf("&", del_1 + 1);
      String data_record = data_packet.substring(del_1 + 1, del_2);
      String selected_class = data_packet.substring(del_2 + 1);

      // Create the request string.
      String request = "?data=OK&mmWave=" + String(data_record)
                     + "&class=" + String(selected_class);
      // Send the obtained mmWave data parameters and the selected pipeline diagnostic class to the web application via an HTTP GET request.
      make_a_get_post_request(false, request);
    }
    
    // If Arduino Nano sends the Run command via serial communication:
    if(data_packet.startsWith("Run")){
      // Glean information as substrings from the transferred data packet by Arduino Nano.
      del_1 = data_packet.indexOf("&");
      String data_record = data_packet.substring(del_1 + 1);
      // Elicit data items from the generated substring.
      del_1 = data_record.indexOf(",");
      del_2 = data_record.indexOf(",", del_1 + 1);
      del_3 = data_record.indexOf(",", del_2 + 1);
      del_4 = data_record.indexOf(",", del_3 + 1);
      del_5 = data_record.indexOf(",", del_4 + 1);
      del_6 = data_record.indexOf(",", del_5 + 1);
      // Convert and store the elicited data items.
      mm.p1 = data_record.substring(0, del_1).toFloat();
      mm.p2 = data_record.substring(del_1 + 1, del_2).toFloat();
      mm.p3 = data_record.substring(del_2 + 1, del_3).toFloat();
      mm.p4 = data_record.substring(del_3 + 1, del_4).toFloat();
      mm.p5 = data_record.substring(del_4 + 1, del_5).toFloat();
      mm.p6 = data_record.substring(del_5 + 1, del_6).toFloat();
      mm.p7 = data_record.substring(del_6 + 1).toFloat();

      // Run the Edge Impulse model to make predictions on the pipeline diagnostic classes.
     run_inference_to_make_predictions(1);

      // Capture a picture with the GC2145 camera.
      take_picture();

      // Create the request string.
      String request = "?results=OK&mmWave=" + String(data_record)
                     + "&class=" + classes[predicted_class];
      // Send the obtained mmWave data parameters, the recently captured image, and the model detection result to the web application via an HTTP POST request.
      make_a_get_post_request(true, request);
    }

    // Clear the received data packet.
    data_packet = "";
  }
}

void run_inference_to_make_predictions(int multiply){
  // Scale (normalize) data items depending on the given model:
  float scaled_p1 = mm.p1;
  float scaled_p2 = mm.p2;
  float scaled_p3 = mm.p3;
  float scaled_p4 = mm.p4;
  float scaled_p5 = mm.p5;
  float scaled_p6 = mm.p6;
  float scaled_p7 = mm.p7;

  // Copy the scaled data items to the features buffer.
  // If required, multiply the scaled data items while copying them to the features buffer.
  for(int i=0; i<multiply; i++){  
    features[feature_ix++] = scaled_p1;
    features[feature_ix++] = scaled_p2;
    features[feature_ix++] = scaled_p3;
    features[feature_ix++] = scaled_p4;
    features[feature_ix++] = scaled_p5;
    features[feature_ix++] = scaled_p6;
    features[feature_ix++] = scaled_p7;
  }

  // Display the progress of copying data to the features buffer.
  Serial.print("Features Buffer Progress: "); Serial.print(feature_ix); Serial.print(" / "); Serial.println(EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE);
  
  // Run inference:
  if(feature_ix == EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE){    
    ei_impulse_result_t result;
    // Create a signal object from the features buffer (frame).
    signal_t signal;
    numpy::signal_from_buffer(features, EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE, &signal);
    // Run the classifier:
    EI_IMPULSE_ERROR res = run_classifier(&signal, &result, false);
    ei_printf("\nrun_classifier returned: %d\n", res);
    if(res != 0) return;

    // Print the inference timings on the serial monitor.
    ei_printf("Predictions (DSP: %d ms., Classification: %d ms., Anomaly: %d ms.): \n", 
        result.timing.dsp, result.timing.classification, result.timing.anomaly);

    // Obtain the prediction results for each label (class).
    for(size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++){
      // Print the prediction results on the serial monitor.
      ei_printf("%s:\t%.5f\n", result.classification[ix].label, result.classification[ix].value);
      // Get the predicted label (class).
      if(result.classification[ix].value >= threshold) predicted_class = ix;
    }
    Serial.print("\nPredicted Class: "); Serial.println(predicted_class);

    // Detect anomalies, if any:
    #if EI_CLASSIFIER_HAS_ANOMALY == 1
      ei_printf("Anomaly : \t%.3f\n", result.anomaly);
    #endif

    // Clear the features buffer (frame):
    feature_ix = 0;
  }
}

void make_a_get_post_request(bool post, String request){
  // Connect to the web application named pipeline_diagnostics_interface. Change '80' with '443' if you are using SSL connection.
  if (client.connect(server, 80)){
    // If successful:
    Serial.println("\nConnected to the web application successfully!\n");
    // Create the query string:
    String query = application + request;
    // Transfer information to the web application via an HTTP POST or GET request depending on the given data parameter type.
    if(post){
      // Make an HTTP POST request:
      String head = "--PipeDetection\r\nContent-Disposition: form-data; name=\"captured_image\"; filename=\"new_image.txt\"\r\nContent-Type: text/plain\r\n\r\n";
      String tail = "\r\n--PipeDetection--\r\n";
      // Get the total message length.
      uint32_t totalLen = head.length() + cam.frameSize() + tail.length();
      // Start the request:
      client.println("POST " + query + " HTTP/1.1");
      client.println("Host: 192.168.1.22");
      client.println("Content-Length: " + String(totalLen));
      client.println("Content-Type: multipart/form-data; boundary=PipeDetection");
      client.println();
      client.print(head);
      client.write(fb.getBuffer(), cam.frameSize());
      client.print(tail);
      client.println("Connection: close");
      client.println();
      // Wait until transferring the image buffer.
      delay(3000);
      // If successful:
      Serial.println("HTTP POST => Data transfer completed!\n");
    }else{
      // Make an HTTP GET request:
      // Start the request:
      client.println("GET " + query + " HTTP/1.1");
      client.println("Host: 192.168.1.22");
      client.println("Connection: close");
      client.println();
      //client.println("Connection: close");
      delay(2000);
      // If successful:
      Serial.println("HTTP GET => Data transfer completed!\n");
    }
  }else{
    Serial.println("\nConnection failed to the web application!\n");
    delay(2000);
  }
}

void take_picture(){
  // Capture a picture with the GC2145 camera.
  // If successful:
  if(cam.grabFrame(fb, 3000) == 0){
   Serial.println("\nGC2145 camera: image captured successfully!");
  }else{
    Serial.println("\nGC2145 camera: image capture failed!");
  }
  delay(2000);
}

AI_assisted_Pipeline_Diagnostics_data_collect.ino

Arduino
         /////////////////////////////////////////////  
        //    AI-assisted Pipeline Diagnostics     //
       //      and Crack Inspection w/ mmWave     //
      //             ---------------             //
     //          (Arduino Nicla Vision)         //           
    //             by Kutluhan Aktar           // 
   //                                         //
  /////////////////////////////////////////////

//
// Export data items from a 60GHz mmWave sensor, train a NN to diagnose pipeline issues, and inspect model results w/ output images on a web app.
//
// For more information:
// https://www.theamplituhedron.com/projects/AI_assisted_Pipeline_Diagnostics_and_Crack_Inspection_w_mmWave/
//
//
// Connections
// Arduino Nano : 
//                                Arduino Nicla Vision
// A0   --------------------------- UART_TX (PA_9) 
// A1   --------------------------- UART_RX (PA_10)
//                                Seeed Studio 60GHz mmWave Sensor
// A2   --------------------------- TX
// A3   --------------------------- RX
//                                2.8'' 240x320 TFT LCD Touch Screen (ILI9341)
// D10  --------------------------- CS 
// D9   --------------------------- RESET 
// D8   --------------------------- D/C
// D11  --------------------------- SDI (MOSI)
// D13  --------------------------- SCK 
// 3.3V --------------------------- LED 
// D12  --------------------------- SDO(MISO)
//                                Control Button (A)
// D2   --------------------------- +
//                                Control Button (B)
// D4   --------------------------- +
//                                Control Button (C)
// D7   --------------------------- +
//                                Control Button (D)
// A4   --------------------------- +
//                                5mm Common Anode RGB LED
// D3   --------------------------- R
// D5   --------------------------- G
// D6   --------------------------- B  


// Include the required libraries:
#include <SoftwareSerial.h>
#include <Adafruit_GFX.h>
#include <Adafruit_ILI9341.h>
#include <60ghzbreathheart.h>

// Define the serial port (Serial1) to communicate with Arduino Nicla Vision via serial communication.
SoftwareSerial Nicla(A0, A1); // RX, TX

// Define the serial port (Serial2) to communicate with the 60GHz mmWave sensor via serial communication.
SoftwareSerial mmWave(A2, A3); // RX, TX

// Define the 60GHz mmWave sensor object.
BreathHeart_60GHz radar = BreathHeart_60GHz(&mmWave);

// Define the required pins for the 240x320 TFT LCD Touch Screen (ILI9341):
#define TFT_CS   10
#define TFT_RST  9
#define TFT_DC   8

// Use hardware SPI (on Nano, SCK, MISO, MOSI) and the above for DC/CS/RST.
Adafruit_ILI9341 tft = Adafruit_ILI9341(TFT_CS, TFT_DC, TFT_RST);

// Define the mmWave radar color scheme.
uint16_t b = ILI9341_BLACK; uint16_t g = ILI9341_GREEN; uint16_t y = ILI9341_YELLOW; uint16_t r = ILI9341_RED;
uint16_t * circle_colors[] = {g,b,r,b,y,b,g,b,r,b,y,b,g,b,r,b,y,b,g,b,r,b,y,b,g,b,r,b,y,b,g,b,r,b,y,b,g,b,r,b,y,b,g,b,r,b,y,b,g,b,r,b,y,b,g,b,r,b,y,b,g,b,r,b,y,b,g,b,r,b,y,b,g,b,r,b,y,b};
// Define the menu button color schemes and names.
uint16_t button_colors[4][2] = {{ILI9341_DARKGREY, ILI9341_BLUE}, {ILI9341_DARKGREY, ILI9341_YELLOW}, {ILI9341_DARKGREY, ILI9341_RED}, {ILI9341_DARKGREY, ILI9341_CYAN}};
String button_names[] = {"A", "B", "C", "D"};

// Define the pipeline diagnostic class names:
String classes[] = {"Leakage", "Cracked", "Clogged"};

// Define the RGB LED pins:
#define redPin     3
#define greenPin   5
#define bluePin    6

// Define the control buttons.
#define button_A   2
#define button_B   4
#define button_C   7
#define button_D   A4

// Define the data holders:
#define TFT_ROTATION  2
String data_packet = "";
volatile boolean command = false;

void setup(){
  Serial.begin(115200);

  // Initialize the software serial ports (Serial1 and Serial2).
  Nicla.begin(115200);
  mmWave.begin(115200);

  pinMode(button_A, INPUT_PULLUP);
  pinMode(button_B, INPUT_PULLUP);
  pinMode(button_C, INPUT_PULLUP);
  pinMode(button_D, INPUT_PULLUP);
  pinMode(redPin, OUTPUT);
  pinMode(greenPin, OUTPUT);
  pinMode(bluePin, OUTPUT);

  // Activate the real-time data transmission mode of the mmWave sensor.
  // radar.reset_func(); delay(1000);
  radar.ModeSelect_fuc(1);
  
  // Initialize the TFT LCD Touch Screen (ILI9341):
  tft.begin();
  tft.setRotation(TFT_ROTATION);
  tft.fillScreen(ILI9341_NAVY);
  tft.setTextColor(ILI9341_WHITE);  tft.setTextSize(2);
  tft.setCursor(10, 10);
  tft.println("Initializing...");
  delay(5000);
  adjustColor(255,0,255);
  
  // Show the mmWave radar and menu buttons.
  int s[4] = {0,0,0,0}; menu_buttons(40,10,5,s,true);
  screen_radar(10);
  delay(1000);
}

void loop(){
  // Collect the data parameters generated by the 60GHz mmWave sensor.
  collect_mmWave_data(true);

  // Send the collected mmWave data and the selected pipeline diagnostic class to Arduino Nicla Vision via serial communication.
  if(!digitalRead(button_A)) { Nicla.print("Data&" + data_packet + "&Leakage"); Serial.println("\nData Sent! Selected Class: Leakage\n"); adjustColor(0,0,255); delay(2000); adjustColor(255,0,255); int s[4] = {1,0,0,0}; menu_buttons(40,10,5,s,false); screen_radar(5); command = true; delay(2000); }
  if(!digitalRead(button_B)) { Nicla.print("Data&" + data_packet + "&Cracked"); Serial.println("\nData Sent! Selected Class: Cracked\n"); adjustColor(255,255,0); delay(2000); adjustColor(255,0,255); int s[4] = {0,1,0,0}; menu_buttons(40,10,5,s,false); screen_radar(5); command = true; delay(2000); }
  if(!digitalRead(button_C)) { Nicla.print("Data&" + data_packet + "&Clogged"); Serial.println("\nData Sent! Selected Class: Clogged\n"); adjustColor(255,0,0); delay(2000); adjustColor(255,0,255); int s[4] = {0,0,1,0}; menu_buttons(40,10,5,s,false); screen_radar(5); command = true; delay(2000); }

  // Send the collected mmWave data parameters to Arduino Nicla Vision via serial communication so as to run the Edge Impulse neural network model.
  if(!digitalRead(button_D)) { Nicla.print("Run&" + data_packet); Serial.println("\nData Parameters Transferred Successfully!\n"); adjustColor(0,255,255); delay(2000); adjustColor(255,0,255); int s[4] = {0,0,0,1}; menu_buttons(40,10,5,s,false); screen_radar(5); command = true; delay(2000); }

  // Undo the menu button selection and clear the latest command.
  if(command){
    int s[4] = {0,0,0,0}; menu_buttons(40,10,5,s,false);
    command = false;
  }
}

void collect_mmWave_data(bool p){
  // Clear the data_packet string.
  data_packet = "";
  
  // Initiate the breath and heartbeat information output.
  radar.Breath_Heart();
  // Add the evaluated breath and heartbeat parameters to the data_packet string.
  if(radar.sensor_report != 0x00){
    if(radar.heart_rate){ data_packet += String(radar.heart_rate, DEC); }else{ data_packet += "0"; }
    if(radar.breath_rate){ data_packet += "," + String(radar.breath_rate, DEC); }else{ data_packet += ",0"; }
  }else{
    data_packet += "0,0";
  }
  delay(500);             

  // Initiate the measuring information output.
  radar.HumanExis_Func();
  if(radar.sensor_report != 0x00){
    if(radar.bodysign_val){ data_packet += "," + String(radar.bodysign_val, DEC); }else{ data_packet += ",0"; }
    if(radar.distance){ data_packet += "," + String(radar.distance, DEC); }else{ data_packet += ",0"; }
    if(radar.Dir_x){ data_packet += "," + String(radar.Dir_x, DEC); }else{ data_packet += ",0"; }
    if(radar.Dir_y){ data_packet += "," + String(radar.Dir_y, DEC); }else{ data_packet += ",0"; }
    if(radar.Dir_z){ data_packet += "," + String(radar.Dir_z, DEC); }else{ data_packet += ",0"; }
  }else{
    data_packet += ",0,0,0,0,0";
  }
  delay(500);

  // Print the collected mmWave data parameters.
  if(p) Serial.println("mmWave Data Parameters: " + data_packet);
}

void screen_radar(int radius){
  int w = tft.width();
  int h = tft.height();
  int x = w/2; int y = w/2;
  int limit = w / (2*radius);
  // Draw the mmWave radar data visualization.
  for(int i=limit; i>0; i--){
    tft.fillCircle(x, y, i*radius, circle_colors[(limit+1)-i]);
    delay(150);
  }
}

void menu_buttons(int a, int e, int offset, int _select[4], bool _init){
  int w = tft.width();
  int h = tft.height();
  int b = (w-(4*a)) / 5;
  int x = b;
  int y = h - a - e;
  // If required, clear the screen.
  if(_init) tft.fillScreen(ILI9341_BLACK);
  // Draw the menu buttons indicating the control button status.
  for(int i=0; i<4; i++){
    tft.fillRect(x+(i*(a+b)), y, a, a, ILI9341_LIGHTGREY);
    tft.fillRect((x+(i*(a+b))+offset), y+offset, a-(2*offset), a-(2*offset), button_colors[i][_select[i]]);
    tft.setTextSize(3);
    tft.setCursor((x+(i*(a+b))+offset+8), y+offset+5);
    tft.println(button_names[i]);
  }
  // Print the activated feature.
  tft.fillRect(0, y-26, w, 25, ILI9341_BLACK);
  tft.setTextSize(2);
  tft.setCursor(20, y-25);
  if(_select[0]) tft.println("Selected: " + classes[0]);
  if(_select[1]) tft.println("Selected: " + classes[1]);
  if(_select[2]) tft.println("Selected: " + classes[2]);
  if(_select[3]) tft.println("EI Model Running!");
}

void adjustColor(int r, int g, int b){
  analogWrite(redPin, (255-r));
  analogWrite(greenPin, (255-g));
  analogWrite(bluePin, (255-b));
}

rgb565_converter.py

Python
from glob import glob
import numpy as np
from PIL import Image

# Obtain all RGB565 buffer arrays transferred by Arduino Nicla Vision as text (.txt) files.
path = "C:\\Users\\kutlu\\New E\\xampp\\htdocs\\pipeline_diagnostics_interface\\detections"
images = glob(path + "/*.txt")

# Convert each RGB565 buffer (TXT file) to a JPG image file and save the generated image files to the images folder.
for img in images:
    loc = path + "/images/" + img.split("\\")[8].split(".")[0] + ".jpg"
    size = (320,320)
    # RGB565 (uint16_t) to RGB (3x8-bit pixels, true color)
    raw = np.fromfile(img).byteswap(True)
    file = Image.frombytes('RGB', size, raw, 'raw', 'BGR;16', 0, 1)
    file.save(loc)
    #print("Converted: " + loc)

class.php

PHP
<?php

// Define the _main class and its functions:
class _main {
	public $conn;
	
	public function __init__($conn){
		$this->conn = $conn;
	}
	
    // Database -> Insert 60GHz mmWave Sensor Data:
	public function insert_new_data($date, $mmWave, $class){
		$sql_insert = "INSERT INTO `entries`(`date`, `mmwave`, `class`) 
		               VALUES ('$date', '$mmWave', '$class');"
			          ;
		if(mysqli_query($this->conn, $sql_insert)){ return true; } else{ return false; }
	}
	
    // Database -> Insert Model Detection Results:
	public function insert_new_results($date, $mmWave, $img, $class){
		$sql_insert = "INSERT INTO `detections`(`date`, `mmwave`, `img`, `class`) 
		               VALUES ('$date', '$mmWave', '$img', '$class');"
			          ;
		if(mysqli_query($this->conn, $sql_insert)){ return true; } else{ return false; }
	}
	
	// Retrieve all data records from the entries database table, transmitted by Arduino Nicla Vision.
	public function get_data_records(){
		$date=[]; $mmWave=[]; $class=[];
		$sql_data = "SELECT * FROM `entries` ORDER BY `id` DESC";
		$result = mysqli_query($this->conn, $sql_data);
		$check = mysqli_num_rows($result);
		if($check > 0){
			while($row = mysqli_fetch_assoc($result)){
				array_push($date, $row["date"]);
				array_push($mmWave, $row["mmwave"]);
				array_push($class, $row["class"]);
			}
			return array($date, $mmWave, $class);
		}else{
			return array(["Not Found!"], ["Not Found!"], ["Not Found!"]);
		}
	}
	
	// Retrieve all model detection results and the assigned detection image names from the detections database table, transferred by Arduino Nicla Vision.
	public function get_model_results(){
		$date=[]; $mmWave=[]; $class=[]; $img=[];
		$sql_data = "SELECT * FROM `detections` ORDER BY `id` DESC";
		$result = mysqli_query($this->conn, $sql_data);
		$check = mysqli_num_rows($result);
		if($check > 0){
			while($row = mysqli_fetch_assoc($result)){
				array_push($date, $row["date"]);
				array_push($mmWave, $row["mmwave"]);
				array_push($class, $row["class"]);
				array_push($img, $row["img"]);
			}
			return array($date, $mmWave, $class, $img);
		}else{
			return array(["Not Found!"], ["Not Found!"], ["Not Found!"], ["icon.png"]);
		}
	}

    // Generate a CSV file from the data records stored in the entries database table.
	public function create_CSV(){
		// Get the stored data records in the entries database table.
		$date=[]; $mmWave=[]; $label=[];
		list($date, $mmWave, $label) = $this->get_data_records();
		// Create the data_records.csv file.
		$filename = "assets/data_records.csv";
		$fp = fopen($filename, 'w');
		// Add the header to the CSV file.
		fputcsv($fp, ["p_1","p_2","p_3","p_4","p_5","p_6","p_7","pipe_label"]);
		// Generate rows from the retrieved data records.
		for($i=0; $i<count($date); $i++){
			$line = explode(",", $mmWave[$i]);
			array_push($line, $label[$i]);
			// Append each generated row to the CSV file as a sample.
			fputcsv($fp, $line);
		}
		// Close the CSV file.
		fclose($fp);
		// Return the CSV file name  data_records.csv.
		return $filename;
	}
}

// Define database and server settings:
$server = array(
	"name" => "localhost",
	"username" => "root",
	"password" => "",
	"database" => "pipeline_diagnostics"
);

$conn = mysqli_connect($server["name"], $server["username"], $server["password"], $server["database"]);

?>

update_server.php

PHP
<?php

include_once "assets/class.php";

// Define the new 'wave' object:
$wave = new _main();
$wave->__init__($conn);

# Get the current date and time.
$date = date("Y_m_d_H_i_s");

# Create the image file name. 
$img_file = "PIPE_".$date;

// If Arduino Nicla Vision sends the collected 60GHz mmWave sensor data with the selected pipeline diagnostic class, save the received information to the entries MySQL database table.
if(isset($_GET["data"]) && isset($_GET["mmWave"]) && isset($_GET["class"])){
	if($wave->insert_new_data($date, $_GET["mmWave"], $_GET["class"])){
		echo "New Data Record Saved Successfully!";
	}else{
		echo "Database Error!";
	}
}

// If Arduino Nicla Vision sends the model detection results, save the received information to the detections MySQL database table.
if(isset($_GET["results"]) && isset($_GET["mmWave"]) && isset($_GET["class"])){
	if($wave->insert_new_results($date, $_GET["mmWave"], $img_file.".jpg", $_GET["class"])){
		echo "Detection Results Saved Successfully!";
	}else{
		echo "Database Error!";
	}
}

// If Arduino Nicla Vision transfers an image of a deformed pipe after running the neural network model, save the received raw image buffer (RGB565) as a TXT file to the detections folder.
if(!empty($_FILES["captured_image"]['name'])){
	// Image File:
	$captured_image_properties = array(
	    "name" => $_FILES["captured_image"]["name"],
	    "tmp_name" => $_FILES["captured_image"]["tmp_name"],
		"size" => $_FILES["captured_image"]["size"],
		"extension" => pathinfo($_FILES["captured_image"]["name"], PATHINFO_EXTENSION)
	);
	
    // Check whether the uploaded file extension is in the allowed file formats.
	$allowed_formats = array('jpg', 'png', 'bmp', 'txt');
	if(!in_array($captured_image_properties["extension"], $allowed_formats)){
		echo 'FILE => File Format Not Allowed!';
	}else{
		// Check whether the uploaded file size exceeds the 5 MB data limit.
		if($captured_image_properties["size"] > 5000000){
			echo "FILE => File size cannot exceed 5MB!";
		}else{
			// Save the uploaded file (image).
			move_uploaded_file($captured_image_properties["tmp_name"], "./detections/".$img_file.".".$captured_image_properties["extension"]);
			echo "FILE => Saved Successfully!";
		}
	}
	
	// Convert the recently saved RGB565 buffer (TXT file) to a JPG image file by executing the rgb565_converter.py file.
	$raw_convert = shell_exec('python "C:\Users\kutlu\New E\xampp\htdocs\pipeline_diagnostics_interface\detections\rgb565_converter.py"');
	print($raw_convert);

	// After generating the JPG file, remove the recently saved TXT file from the server.
	unlink("./detections/".$img_file.".txt");
}

// If requested, create a CSV file from the data records saved in the entries database table.
if(isset($_GET["create_CSV"])){
	// Create the data_records.csv file.
	$filename = $wave->create_CSV();
	// Download the generated CSV file automatically.
    header('Content-Description: File Transfer');
    header('Content-Type: application/octet-stream');
    header("Cache-Control: no-cache, must-revalidate");
    header('Content-Disposition: attachment; filename="'.basename($filename).'"');
    header('Content-Length: '.filesize($filename));
	header('Pragma: public');
    readfile($filename);
}

?>

show_records.php

PHP
<?php

include_once "assets/class.php";

// Define the new 'wave' object:
$wave = new _main();
$wave->__init__($conn);

// Obtain all data records from the entries database table and print them as table rows.
$date=[]; $mmWave=[]; $label=[];
list($date, $mmWave, $label) = $wave->get_data_records();
$records = "<tr><th>Date</th><th>mmWave</th><th>Label</th></tr>";
for($i=0; $i<count($date); $i++){
	$records .= '<tr class="'.$label[$i].'">
				  <td>'.$date[$i].'</td>
				  <td style="word-break:break-all;">'.$mmWave[$i].'</td>
				  <td>'.$label[$i].'</td>
			    </tr>
			    ';   
}

// Fetch all model detection results with the assigned detection image name from the detections database table and display them as table rows.
$date_R=[]; $mmWave_R=[]; $class=[]; $img=[];
list($date_R, $mmWave_R, $class, $img) = $wave->get_model_results();
$results = "<tr><th>Date</th><th>mmWave</th><th>Model Prediction</th><th>IMG</th></tr>";
for($i=0; $i<count($date_R); $i++){
	$results .= '<tr class="'.$class[$i].'">
				  <td>'.$date_R[$i].'</td>
				  <td style="word-break:break-all;">'.$mmWave_R[$i].'</td>
				  <td>'.$class[$i].'</td>
				  <td><img src="detections/images/'.$img[$i].'"/></td>
			    </tr>
			    ';   
}

// Create a JSON object from the generated table rows consisting of the obtained data records and model detection results.
$result = array("records" => $records, "results" => $results);
$res = json_encode($result);

// Return the recently generated JSON object.
echo($res);

?>

index.php

PHP
<!DOCTYPE html>
<html>
<head>
<title>AI-assisted Pipeline Diagnostics and Crack Inspection</title>

<!--link to index.css-->
<link rel="stylesheet" type="text/css" href="assets/index.css"></link>

<!--link to favicon-->
<link rel="icon" type="image/png" sizes="36x36" href="assets/icon.png">

<!-- link to FontAwesome-->
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v6.2.1/css/all.css">
 
<!-- link to font -->
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Playfair+Display+SC:ital@1&display=swap" rel="stylesheet">

<!--link to jQuery script-->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script>

</head>
<body>
<?php ini_set('display_errors',1);?> 
<h1><i class="fa-solid fa-faucet-drip"></i> AI-assisted Pipeline Diagnostics and Crack Inspection w/ mmWave</h1>
<div class="results">
<table>
<tr><th>Date</th><th>mmWave</th><th>Model Prediction</th><th>IMG</th></tr>
<tr><td>X</td><td>X</td><td>X</td><td><img src="assets/icon.png" /></td></tr>
</table>
</div>

<div class="records">
<button class="create"><i class="fa-solid fa-file-csv fa-beat"></i> Create CSV</button>
<table>
<tr><th>Date</th><th>mmWave</th><th>Label</th></tr>
<tr><td>X</td><td>X</td><td>X</td></tr>
</table>
</div>

<!--Add the index.js file-->
<script type="text/javascript" src="assets/index.js"></script>

</body>
</html>

index.js

JavaScript
// If requested, create a CSV file (data_records.csv) from the data records saved in the entries database table.
$(".records").on("click", "button", () => {
	if(confirm(" Download the generated CSV file!\n\n data_records.csv")){
		window.location = "./update_server.php?create_CSV=OK";
	}
});

// Every 5 seconds, retrieve the HTML table rows generated from the database table rows to display the latest collected data records
// and inform the user of the most recent model detection results on pipeline diagnostic classes.
setInterval(function(){
	$.ajax({
		url: "./show_records.php",
		type: "GET",
		success: (response) => {
			// Decode the obtained JSON object.
			const res = JSON.parse(response);
			// Assign the data record HTML table rows.
			$(".records table").html(res.records);
			// Assign the model detection HTML table rows.
			$(".results table").html(res.results);
		}
	});
}, 5000);

index.css

CSS
html{background-image:url('background.jpg');background-repeat:no-repeat;background-attachment:fixed;background-size:100% 100%;font-family: 'Playfair Display SC', serif;}
h1{position:fixed;top:2%;right:30px;text-align:right;font-weight:bold;padding-right:15px;user-select:none;width:380px;background:-webkit-linear-gradient(45deg, #D1CDDA, #FAD7BD);-webkit-background-clip:text;-webkit-text-fill-color:transparent;}


.results{position:fixed;bottom:5.5%;left:15px;width:55%;height:85%;background-color:rgba(37, 40, 42, 0.6);overflow-y:auto;border:10px solid rgba(37, 40, 42, 0.8);border-radius:20px;padding:5px;}
.results table{position:relative;width:95%;color:white;margin:auto;margin-top:20px;margin-bottom:20px;border:3px solid #FAD7BD;user-select:none;}
.results th, .results td{border:3px solid #FAD7BD;color:#25282A;}
.results th{background-color:#FAD7BD}
.results td{color:white;padding:10px;}
.results img{display:block;width:120px;height:120px;margin:auto;border:3px solid #FAD7BD;border-radius:5px;padding:5px;transition:1s;}
.results img:hover{cursor:crosshair;border:3px solid orange;padding:10px;transition:1s;}

.Leakage{background-color:rgba(4, 52, 88, 0.2);}
.Cracked{background-color:rgba(184, 175, 230, 0.3);}
.Clogged{background-color:rgba(226, 54, 54, 0.2);}

.records{position:fixed;bottom:5.5%;right:15px;width:38%;height:45%;background-color:rgba(37, 40, 42, 0.6);overflow-y:auto;border:10px solid rgba(37, 40, 42, 0.8);border-radius:20px;padding:5px;}
.records table{position:relative;width:95%;color:white;margin:auto;margin-bottom:20px;border:3px solid #FAD7BD;user-select:none;}
.records th, .records td{border:3px solid #FAD7BD;color:#25282A;}
.records th{background-color:#FAD7BD;}
.records td{color:white;padding:10px;}
.records button{display:block;margin:auto;margin-bottom:10px;margin-top:10px;background-image:linear-gradient(45deg, #D1CDDA, #FAD7BD);width:80%;height:45px;font-size:24px;border:2px solid white;border-radius:15px;font-weight:bold;color:white;}
.records button:hover{cursor:pointer;background-image:linear-gradient(45deg, #FAD7BD, #FAD7BD);}


/* Width */
::-webkit-scrollbar {width:5px;height:5px;}
/* Track */
::-webkit-scrollbar-track {background-color:#25282A;}
/* Button */
::-webkit-scrollbar-button{background-color:#FAD7BD;height:5x;width:5px;}
::-webkit-scrollbar-button:hover{background-color:white;}
/* Handle */
::-webkit-scrollbar-thumb {background-color:#E4E4E2;}
::-webkit-scrollbar-thumb:hover {background-color:white;}
/* Corner */
::-webkit-scrollbar-corner{background-color:#F5DDC5;}

Credits

Kutluhan Aktar

Kutluhan Aktar

81 projects • 305 followers
AI & Full-Stack Developer | @EdgeImpulse | @Particle | Maker | Independent Researcher

Comments

Related channels and tags
  • 3D Printing
  • Anycubic
  • Arduino
  • Autodesk
  • Edge Impulse
  • Elecrow
  • Internet of Things
  • KiCad
  • Machine Learning & AI
  • Microsoft
  • PCBWay
  • Seeed Studio
  • SparkFun