The project "Pipe 360: Urban Pipe Predictive Freeze Alert System" was born out of the necessity to tackle a common but often overlooked problem in urban areas: the freezing of water pipes during cold seasons. This issue can lead to significant water loss, property damage, and costly repairs for city management and residents alike. Recognizing the impact of climate variability and the increasing occurrence of extreme weather conditions, the project aims to offer a proactive solution to monitor, predict, and alert communities about potential freeze risks in water piping systems.
At the heart of Pipe 360 lies a combination of Internet of Things (IoT) technology and Machine Learning (ML) algorithms, designed to gather real-time data on temperature, water flow, and environmental conditions around the pipe infrastructure. This data is then analyzed to predict the likelihood of pipe freezing, allowing for timely alerts and actions to prevent damage. By integrating with Microchip's AVR-IoT Cellular Mini, the system ensures reliable connectivity and data transmission to a central server, where ML models assess the risk of freezing based on the collected sensor data.
The decision to develop Pipe 360 stemmed from a desire to leverage technology for urban resilience, enhancing the preparedness and adaptive capacity of cities to face environmental challenges. It represents a step forward in smart city initiatives, emphasizing the importance of predictive maintenance and sustainable infrastructure management.
The project not only addresses an immediate and practical need but also demonstrates how innovative tech solutions can be applied to improve urban living conditions, reduce economic losses, and contribute to environmental sustainability. Through Pipe 360, we envision a future where urban communities are better equipped to handle the vagaries of weather, ensuring a reliable and safe water supply for all residents.
Flow Diagram of the projectBattery: The system begins with a battery that powers the entire setup, ensuring it can operate independently without the need for direct power from the grid.
Water Flow Sensor 5V: This sensor is connected to the battery and monitors the flow of water through the pipes. It is essential for detecting stagnation, which can contribute to freezing conditions.
- Specifications:
Frequency: F=7.5 * Q (L / Min) error: ± 2%, current can not exceed 10mA, 450 output pulses/liters. F=onstant * units of flow (L / min) * time (seconds)
Flow range:1-30L/min
AVR IoT Cellular Mini: This is the heart of the setup. It receives input from the water flow sensor and the MCP9808 temperature sensor. It's responsible for processing this data and communicating with Flask Server.
MCP9808: This is a precise temperature sensor that feeds temperature data to the AVR IoT Cellular Mini. Monitoring temperature is crucial for predicting potential freezing in the pipes.
HTTP_client POST call: The AVR IoT Cellular Mini makes HTTP POST requests to send temperature and water flow readings to Flask Server. This happens every 600 seconds (or 10 minutes), providing regular updates for analysis.
Server: The server receives the POST request and processes the data. It has an endpoint that accepts these requests and can handle GET requests for data retrieval as well.
Random Forest Classifier: Server uses a machine learning model, specifically a Random Forest Classifier, to predict freeze conditions based on incoming data.
Alerting System: If the model predicts a freeze condition, the server triggers an alerting system. In this case, it sends out email alerts to warn of potential freezing conditions.
Machine Learning Flow (Right Side of Diagram):
- Data Gathering: The first step in the machine learning workflow involves collecting data, likely historical data on pipe temperatures, flow rates, and freeze incidents.
- Data Analysis: This step involves examining the gathered data to understand patterns and features that can influence the predictions.
- Train: Train the Random Forest Classifier using the analyzed data, learning to predict pipe freezing conditions.
- Test: After training, test the model to evaluate its accuracy and performance.
- Save Pickle File: Once the model is trained and tested, save it as a pickle file, which is a way of serializing the model so it can be loaded and used directly without retraining.
Email Alerts: This is the last step in the process, where the server sends an email alert if the model predicts a freeze condition, providing timely warnings to take preventative action.
Circuit ConnectionTo ensure seamless integration and independence of the IoT system, I've established my own server, which is configured to communicate with the AVR IoT Cellular Mini device. I achieved this by setting up port forwarding rules that allow the IoT device to send data directly to the server over the internet. This configuration is crucial as it ensures that the device can reliably transmit sensor readings at regular intervals without the need for a direct connection to a computer or manual intervention.
Logs from the server
<Request 'http://<server address>:5000/log' [POST]>
Received data: b'{"Temperature":28.1875,"Water Flow":2}'
Email sent successfully!
<Request 'http://<server address>:5000/log' [POST]>
Received data: b'{"Temperature":27.875,"Water Flow":0}'
<ip> - - [03/Mar/2024 20:31:09] "POST /log HTTP/1.1" 200 -
<Request 'http://<server address>:5000/log' [POST]>
Received data: b'{"Temperature":28.1875,"Water Flow":1.2}'
<ip> - - [03/Mar/2024 20:38:48] "POST /log HTTP/1.1" 200 -
<Request 'http://<server address>:5000/log' [POST]>
Received data: b'{"Temperature":28.25,"Water Flow":1.2}'
<ip> - - [03/Mar/2024 20:40:46] "POST /log HTTP/1.1" 200 -
<Request 'http://<server address>:5000/log' [POST]>
Received data: b'{"Temperature":28.75,"Water Flow":1.5}'
<ip> - - [03/Mar/2024 20:41:42] "POST /log HTTP/1.1" 200 -
<Request 'http://<server address>:5000/log' [POST]>
Received data: b'{"Temperature":28.75,"Water Flow":1.5}'
<ip> - - [03/Mar/2024 20:45:49] "POST /log HTTP/1.1" 200 -
By examining the server logs, I can confirm that the device is consistently posting data from the sensors. This hands-off approach not only streamlines the monitoring process but also solidifies the system’s capability to function autonomously, providing real-time insights into the temperature and water flow within the urban pipe infrastructure. The independent nature of this setup is pivotal for the proactive and predictive maintenance capabilities of the project.
Logs from Microchip IOT
- Temperature (°C): Measures the ambient temperature around the pipe which directly influences the freezing risk.
- Water Flow (L/min): Indicates the rate at which water is flowing through the pipe, with lower rates increasing the likelihood of freezing.
- Pipe Material: The type of material the pipe is made from, which affects its thermal properties and susceptibility to freezing.
- Pipe Insulation: The level of insulation around the pipe, crucial for retaining heat and preventing freezing.
- Pipe Diameter (cm): The width of the pipe, with smaller diameters potentially freezing faster due to a lower volume of water.
- Pipe Length (m): The length of the pipe section being monitored; longer pipes may have different freezing characteristics.
- Exposure to Wind: The degree to which the pipe is exposed to wind, which can increase heat loss and the freezing risk.
- Humidity (%): The amount of moisture in the air around the pipe, which can affect the thermal exchange and freezing process.
- Pipe Location: Where the pipe is situated, such as in unheated areas like basements or attics, which are more prone to freezing.
- Air Circulation: The flow of air around the pipe, with poor circulation potentially leading to colder pipe surfaces and higher freezing risk.
- True Negatives (TN): 18801 instances were correctly predicted as non-freezing conditions.
- False Positives (FP): 17 instances were wrongly predicted as freezing conditions (type I error).
- False Negatives (FN): 1054 instances were wrongly predicted as non-freezing conditions (type II error).
- True Positives (TP): 128 instances were correctly predicted as freezing conditions.
Accuracy is the ratio of the correctly predicted instances to the total instances
approximately 94.645% of the predictions made by the model are correct
Alerting SystemThe email alerting system in this project is designed to notify the user when the machine learning model predicts a freeze condition. It uses the Simple Mail Transfer Protocol (SMTP) to send out email alerts through Gmail, one of the most commonly used email services.
Running the Python Server Open Terminal or Command Prompt: Navigate to the folder containing your server Python script (server.py).
Run the Server: Execute the Python server script by running the command:
python server.py
This will start the Flask server, listening for incoming HTTP requests from your AVR-IoT Cellular Mini device.
Training the Machine Learning ModelPrepare Your Data: Ensure your dataset is ready and accessible in the same directory as your training script or specify the path to it within the script.
Open a New Terminal or Command Prompt Window: Navigate to the folder containing your training Python script (train.py).
Run the Training Script: Execute the training script by running the command:
python train.py
This script will train your machine learning model using the provided dataset and save the trained model to a file (e.g., model.pkl).
Running the Arduino Code on AVR DB Series Board Open the Arduino IDE: Launch the Arduino Integrated Development Environment (IDE) on your computer.
Load the.ino File: Navigate to File > Open, then find and select the.ino file you wish to upload. This will open the file in the IDE.
Configure the IDE for AVR DB Series:
Select the Board: Go to Tools > Board: "" > and select "AVR DB-series" from the list. If you do not see the AVR DB-series option, you may need to install or update the board definitions in your Arduino IDE by going to Tools > Board > Boards Manager..., searching for "DxCore" (which supports AVR DB-series), and installing or updating it. Choose the Correct Serial Port: Go to Tools > Port and select the port that your AVR DB-series board is connected to. If unsure, disconnect your device, open the menu to note the available ports, reconnect your device, and select the newly appeared port. Upload the Code: Press the Upload button (the right arrow icon) within the IDE. This action compiles the Arduino sketch and uploads it to your AVR DB-series board. Wait for the process to complete, indicating that your device is now running the uploaded sketch.
Monitor the Output (Optional): To view debug output or messages from your device, open the Serial Monitor within the Arduino IDE by clicking the magnifying glass icon in the upper-right corner. Ensure the baud rate in the Serial Monitor matches the baud rate specified in your sketch (e.g., Serial.begin(115200);).
Make sure the your settings in IDE matches below
Training a ML Model
1. Imports and Data Generation
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import LabelEncoder
import pickle
Libraries are imported for handling data (pandas
, numpy
), machine learning model training and evaluation (sklearn
), and saving the trained model (pickle
).
np.random.seed(42)
n_samples = 100000
data = {...}
df = pd.DataFrame(data)
A dataset with 100, 000 samples is generated, covering various conditions like temperature, water flow, pipe material, etc. This dataset is converted into a pandas DataFrame for easy manipulation
2. Feature Engineering and Noise Introduction
def determine_freezing_condition_adjusted(row):
if row['Temperature (°C)'] <= 0 and row['Water Flow (L/min)'] < 0.5 and \
row['Pipe Insulation'] in ['None', 'Low'] and \
row['Pipe Location'] in ['Basement', 'Attic', 'Outside Wall'] and \
row['Air Circulation'] == 'Poor':
return 1
else:
return 0
df['Freeze Condition'] = df.apply(determine_freezing_condition_adjusted, axis=1)
A custom function checks each row for specific conditions that lead to pipe freezing and labels those rows accordingly.
noise_level = 0.05
df['Temperature (°C)'] += np.random.normal(0, noise_level * np.abs(df['Temperature (°C)'].mean()), n_samples)
df['Water Flow (L/min)'] += np.random.normal(0, noise_level * np.abs(df['Water Flow (L/min)'].mean()), n_samples)
Noise is added to Temperature (°C)
and Water Flow (L/min)
to simulate measurement errors, using a normal distribution centered around 0 with a standard deviation
flip_percentage = 0.05
indices_to_flip = np.random.choice(df.index, size=int(flip_percentage * len(df)), replace=False)
df.loc[indices_to_flip, 'Freeze Condition'] = 1 - df.loc[indices_to_flip, 'Freeze Condition']
A small percentage (5%) of the 'Freeze Condition' labels are flipped to introduce label noise, simulating inaccuracies in the data labeling process
3. Data Preprocessing, Model Training, and Serialization
encoder = LabelEncoder()
for col in ['Pipe Material', 'Pipe Insulation', 'Exposure to Wind', 'Pipe Location', 'Air Circulation']:
df[col] = encoder.fit_transform(df[col])
Categorical features are encoded into numerical format using LabelEncoder
X = df.drop('Freeze Condition', axis=1)
y = df['Freeze Condition']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
The DataFrame is split into features (X) and the target variable (y), then divided into training and test sets with 20%
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
predictions = model.predict(X_test)
accuracy = accuracy_score(y_test, predictions)
print(f'Accuracy: {accuracy}')
4. Why RandomForest ML Model?
Random Forest is generally robust to noise and less prone to overfitting.The dataset likely contains complex interactions among variables that affect whether a pipe will freeze under certain conditions. Random Forest can capture these non-linear relationships without the need for explicit feature engineering, making it well-suited for datasets with complex interactions between features.
Its relatively low need for hyperparameter tuning make it an excellent choice for predicting pipe freezing conditions based on various environmental and physical parameters.
5. Saving ML Model
with open('piper.pkl', 'wb') as file:
pickle.dump(model, file)
Finally, the trained model is saved to a file using pickle. This allows the model to be loaded and used later without the need for retraining.
Setting up the Server1.Importing Libraries
from flask import Flask, request, jsonify
from datetime import datetime, timedelta
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
import pickle
import numpy as np
import json
app = Flask(__name__)
- Libraries for web server functionality, sending emails, handling dates, and processing data are imported.
- A Flask web server instance is initialized to handle incoming web requests.
2. Loading the Machine Learning Model
model_path = './piper.pkl'
with open(model_path, 'rb') as file:
model = pickle.load(file)
The trained machine learning model is loaded from a file. This model is used later to predict pipe freeze conditions based on incoming data.
3. Sending Email Alerts
def send_email(subject, body):
sender_email = "<Sender Email>"
receiver_email = "<Receiver Email>"
password = "<App Password>"
message = MIMEMultipart()
message["From"] = sender_email
message["To"] = receiver_email
message["Subject"] = subject
detailed_body = """ <Detailed Alert Message present in the linked github code>"""
message.attach(MIMEText(detailed_body, "plain"))
try:
with smtplib.SMTP_SSL("smtp.gmail.com", 465) as server:
server.login(sender_email, password)
server.sendmail(sender_email, receiver_email, message.as_string())
print("Email sent successfully!")
except Exception as e:
print(f"Failed to send email: {e}")
This function defines how to send an email alert. It creates an email with a detailed body including precautions to prevent pipe freezing, using the smtplib
library for email sending.It uses Python's smtplib
library to connect securely to Gmail's SMTP server.
4. Logic for Sending Freeze Email Alerts
try:
...
except KeyError:
return jsonify({"status": "error", "message": "Missing data in request"}), 400
features = np.array([...]).reshape(1, -1)
prediction = model.predict(features)
current_time = datetime.now()
if prediction[0] == 1 and (last_alert_time is None or current_time - last_alert_time >= timedelta(hours=1)):
last_alert_time = current_time
send_email("Freeze Alert", "A freeze condition is likely to occur.")
response_message = "Freeze alert sent."
else:
response_message = "No alert sent."
- This logic checks for required fields in the received data. If any are missing, it returns an error response.
- It prepares the data for the model, makes a prediction, and then, based on the prediction and the time elapsed since the last alert, decides whether to send a new alert.
- An email is sent if a freeze condition is predicted and at least one hour has passed since the last alert.
5.Server Health Check Endpoint
@app.route('/', methods=['GET'])
def hello():
return jsonify({"status": "success", "message": "Welcome to the Freeze Alert System"}), 200
This route responds to GET requests at the root URL (/
), It serves as a basic health check
6. Using Server
POST Endpoint for Receiving Log Data from Microchip
- Endpoint:
/log
- Method: POST
curl -X POST http://<your-server-address>:5000/log -d '{"Temperature": 5, "Water Flow": 0.4}' -H "Content-Type: application/json"
GET Endpoint for Server Health Check
- Endpoint: /
- Method: GET
curl http://<your-server-address>:5000/
Dive into Arduino codeIncluding Libraries and Global Variables
#include <mcp9808.h>
#include <Arduino.h>
#include <http_client.h>
#include <ArduinoJson.h>
#include <led_ctrl.h>
#include <log.h>
#include <lte.h>
const char* serverPath = "/log";
const char* serverDomain = "<Your Server address>";
const int serverPort = 5000;
Libraries for temperature sensing (mcp9808
), HTTP communication (http_client
), JSON handling (ArduinoJson
), LED control (led_ctrl
), logging (log
), and LTE connectivity (lte
) are included, highlighting the diverse functionality of the AVR-IoT Cellular Mini.
Configuration for server communication (path, domain, port)
const byte flowSensorPin = 13;
const float flowCalibrationFactor = 0.2;
volatile byte flowPulseCount = 0;
float flowRate = 0.0;
unsigned long oldTime = 0;
void flowPulseCounter() {
flowPulseCount++;
}
This initializes flow sensor (pin, calibration factor, pulse count, rate, timing) are set up. These globals are crucial for data collection and transmission.
void sendDataToServer(float temperature, float flowRate) {
if (!HttpClient.configure(serverDomain, serverPort, false)) {
Log.error("Failed to configure HTTP client");
return;
}
JsonDocument doc;
doc["Temperature"] = temperature;
doc["Water Flow"] = flowRate;
String json;
serializeJson(doc, json);
HttpResponse response = HttpClient.post(serverPath, json.c_str());
if (response.status_code > 0) {
Log.infof("POST - status code: %u, data size: %u\r\n", response.status_code, response.data_size);
} else {
Log.error("Error on sending POST");
}
}
JsonDocument
object is created to store "Temperature" and "Water Flow".The JSON payload is converted to a C-style string (json.c_str()
) and passed as the data for the POST request. The response from the server is stored in a variable named response
.
Setup Explain
void setup() {
Serial.begin(115200);
Log.begin(115200);
LedCtrl.begin();
LedCtrl.startupCycle();
const int8_t error = Mcp9808.begin();
if (error) {
Serial.println("Error: could not start MCP9808 library");
}
if (!Lte.begin()) {
Log.error("Failed to connect to the operator");
return;
}
pinMode(flowSensorPin, INPUT_PULLUP);
attachInterrupt(digitalPinToInterrupt(flowSensorPin), flowPulseCounter, FALLING);
Log.info("Flow sensor measurement started");
Log.infof("Connected to operator: %s\r\n", Lte.getOperator().c_str());
}
Setting a baud rate of 115200 bits per second, kick off serial communication.LEDs are initialized and go through a startup cycle.The device attempts to start the MCP9808 temperature sensor.
if (!Lte.begin()) {
Log.error("Failed to connect to the operator");
return;
}
This part attempts to connect to a cellular network.
The device sets up a digital pin to read pulses from a flow sensor, using interrupts to catch these pulses efficiently.The device logs the start of flow measurement and confirms a successful connection to the cellular network
float readFlowRate() {
float flowRateLPM = 0.0;
if ((millis() - oldTime) > 900) {
detachInterrupt(digitalPinToInterrupt(flowSensorPin));
flowRateLPM = (flowPulseCount / flowCalibrationFactor) / ((millis() - oldTime) / 60000.0);
oldTime = millis();
flowPulseCount = 0;
attachInterrupt(digitalPinToInterrupt(flowSensorPin), flowPulseCounter, FALLING);
}
return flowRateLPM;
}
The function initiates with a presumption that the flow rate is zero (no flow). The function checks the time elapsed since it last calculated the flow rate. If more than 900 milliseconds have passed, it proceeds to calculate a new flow rate. This delay ensures that enough time has passed to accumulate a measurable amount of data
attachInterrupt(digitalPinToInterrupt(flowSensorPin), flowPulseCounter, FALLING);
Before calculating the flow rate, the function temporarily stops listening for new pulses.
flowRateLPM = (flowPulseCount / flowCalibrationFactor) / ((millis() - oldTime) / 60000.0);
The function calculates the flow rate based on the number of pulses counted, adjusted by a calibration factor to account for the specific sensor's characteristics. It divides this by the time elapsed since the last calculation, converting milliseconds to minutes to get the rate in liters per minute.
void loop() {
const float temperature = Mcp9808.readTempC();
flowRate = readFlowRate();
Serial.printf("Temperature (*C): %f\r\n", (double)temperature);
sendDataToServer(temperature, flowRate);
delay(600000);
}
First, the temperature is read from the MCP9808 sensor. This sensor is known for its accuracy and is used here to measure the ambient temperature.Next, the function calculates the flow rate using the readFlowRate()
function detailed previously. This value reflects the current rate at which fluid is passing through the flow sensor, measured in liters per minute (LPM)
The function pauses for 600, 000 milliseconds, or 10 minutes. This delay is a power and resource management decision, balancing the need for timely data against the desire to conserve battery life and reduce data transmission volume.
ConclusionIn summary, "Pipe 360" embodies the cutting-edge fusion of IoT and machine learning, offering a forward-thinking solution to urban infrastructure protection. Central to this innovation is the Microchip AVR-IoT Cellular Mini, which seamlessly bridges the gap between local data collection and global internet connectivity. By leveraging this device's robust capabilities, "Pipe 360" represents a significant step toward smarter, more resilient cities by preemptively addressing the critical issue of pipe freezing. This system not only anticipates potential disruptions but also equips residents with timely alerts, allowing for prompt preventative measures. As a testament to innovation in urban planning and the pivotal role of advanced IoT technology, "Pipe 360" sets a new standard for community safeguarding and infrastructure longevity.
Comments