Published September 30, 2021 © GPL3+

Don't let the mice in, close the garage door!

At the end of the day, I will get a text and email (with photo) indicating if a garage door is opened or closed.

IntermediateFull instructions provided10 hours75

Don't let the mice in, close the garage door!

Things used in this project

Hardware components

Google AIY Vision

I just happened to repurpose this kit but you can you anything that has a camera and can do image classification.

smart plug

Software apps and online services

Google Colab

Twilio SMS Messaging API

Story

My wife was getting frustrated with me continually leaving the garage door open. Mice would get in overnight, and we would find them in traps in the house. Sure, I could get an inexpensive Wifi adapter for my garage door openers and remotely close them via an app, but what fun is that? I figured I could put my newly purchased Google AIY Vision kit to use and see what I could do with computer vision!

For those that want to get familiar with the Vision Kit, I recommend going to https://aiyprojects.withgoogle.com/vision to see the kit and the examples. It consists of a Raspberry Pi Zero WH, a Raspberry Pi V2 Camera, and a Vision Bonnet (a Myriad 2450 ML accelerator) along with some other stuff (LEDs, buzzer). After experimenting with the smile detector and other examples, I wanted to put the kit to use.

I set up the kit on a small shelf that I installed in my garage facing towards the garage doors. Totally optional, but I plugged the kit into a smart plug, so it could power down at night and start up in the morning (what's the point of it running overnight?).

Google AIY Vision Kit set up in the garage.

First, I needed to collect some data for the computer vision model. I wrote a simple Python script to take a photo and save it with a timestamp. I then set up a cron job run that script once an hour from sunrise to sunset. I did this for about a week to get a good range of lighting and different states (one door open, both doors open, both doors closed, garage empty, 1 or 2 cars in the garage) for the model. Python script code below:

#!/usr/bin/env python3
#Take some photos of the garage to further improve the image classifier
#Run hourly as a cron job during daylight hours

from datetime import datetime
from picamera import PiCamera, Color


#set the filename path for the garage photos
timenow = datetime.now()
timeStr = timenow.strftime("%m-%d-%Y-%H%M%S")
file_path = '/home/pi/garage_screenshots/garage_' + timeStr + '.jpg'

with PiCamera(sensor_mode=4, resolution=(1640, 1232), framerate=30) as camera:
    camera.capture(file_path)

Once I had enough photos, I leveraged the Google Colab from the Vision Kit tutorial. It runs a default flower classification model so I had to replace the flower photos with the photos from my garage. Given that a Colab is only a temporary instance, you have to upload the photos to the Colab each time you run it. A small price to pay for being able to use the Google GPUs to train your model. I created a garage_photos folder with 2 subfolders: open and closed. These subfolders align to the inference labels that you would get as output from the model. I populated the open folder with photos with at least one garage door open, and populated the closed folder with photos of both the doors closed Here is the change I made in the Colab:

python scripts/retrain.py \
--bottleneck_dir=tf_files/bottlenecks \
--how_many_training_steps=500 \
--model_dir=tf_files/models/ \
--summaries_dir=tf_files/training_summaries/$ARCHITECTURE \
--output_graph=tf_files/retrained_graph.pb \
--output_labels=tf_files/retrained_labels.txt \
--architecture=$ARCHITECTURE \
--image_dir=tf_files/garage_photos

All the other steps remain the same. I saved a local copy of the Google Colab to my Google Drive so I wouldn't have to update the Colab each time I ran it. Based on the output, my model was getting low 90s in accuracy for validation and test data. Good enough for me, and I could continually add more photos to improve the model results.

Once the Colab was complete, I downloaded the model (two files are created: retrained_graph.binaryproto and retrained_labels.txt) and copied them over to the Raspberry Pi Zero.

Now, the script to check the garage door status! I kept this simple as well, adding a cron job to run the script only once (near dusk, so I change the time as we get more or less sunlight). I *heavily* leveraged Google's mobilenet_based_classifier.py Python script to feed in the model arguments and run inference. Once the results were in, I took a camera capture (just to validate the inference results) and emailed the garage door status and a photo to my wife and I using smtplib. I also set up a Twilio trial account to send a text message of the garage door status (no image though, to keep it cheaper) using the Twilio API. And when the cronjob ran, success!

A screen capture of the emails that I receive at the end of the day.

Screenshot of the text messages we get.

Future improvements?

So what's next? I think the next step for me is to automate it! If the garage door is open, I still have to go downstairs and close it. How much of a pain is that? :) Why not just have the program close the garage door if it detects that it is open?

That will be version 2 of this project. I started testing out a 2-channel relay switch with the vision kit, and I was able to run a simple script to have each relay open and close (but boy was it a pain to figure out how to wire it given that it is a 5V relay with 3.3V logic on the Raspberry Pi!). I plan to run wire from the relay to the push buttons for the garage door openers and trigger them when the program detects a door open. I will also have to update the model to differentiate between the two garage doors. Right now my model only cares if 1 door is open or if they are both closed.

Version 2 involves a 2 channel relay!

I hope you enjoyed this project. It was fun to learn the inner workings of the Vision Kit and then repurpose it for something practical for my home. Feel free to leave me any comments and/or feedback. Thanks!

Code

Garage Image Classifier Python code

#!/usr/bin/env python3
#
# Copyright 2017 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Script to run generic MobileNet based classification model."""
import argparse
#Justin is hijacking this code to develop a program to identify if the garage door is open or not
#model name is retrained_graph3.binaryproto

from picamera import PiCamera, Color
import numpy as np

from aiy.vision import inference
from aiy.vision.models import utils

from twilio.rest import Client
import os

import smtplib
from email.mime.text import MIMEText
from email.mime.image import MIMEImage
from email.mime.multipart import MIMEMultipart

HOSTNAME = smtp_server_here
USERNAME = 'email_addr'
PASSWORD = 'pword'
RECIPIENT_ADDRESS = ['xxxxxxx@mail.com', 'xxxxxx@mail.com']

def sendMail(file_str, SUBJECT):
	EMAIL_BODY = SUBJECT
	
	msg = MIMEMultipart('related')
	msg['Subject'] = SUBJECT
	msg['From'] = USERNAME
	msg['To'] = ", ".join(RECIPIENT_ADDRESS)

	html = """\
	<html>
	  <head></head>
		<body>
		  <img src="cid:image1" alt="Logo" style="width:600px;height:400px;"><br>
		   <p><h4 style="font-size:15px;">Here is the current garage door status.</h4></p>           
		</body>
	</html>
	"""
	# Record the MIME types of text/html.
	part2 = MIMEText(html, 'html')

	# Attach parts into message container.
	msg.attach(part2)

	fp = open(file_str, 'rb')
	msgImage = MIMEImage(fp.read())
	fp.close()

	msgImage.add_header('Content-ID', '<image1>')
	msg.attach(msgImage)

	server = smtplib.SMTP(HOSTNAME, 587)
	server.ehlo()
	server.starttls()
	server.login(USERNAME, PASSWORD)	
	server.sendmail(USERNAME, RECIPIENT_ADDRESS, msg.as_string())
	server.quit()
	print ("Email sent. Logging Out...")

def read_labels(label_path):
    with open(label_path) as label_file:
        return [label.strip() for label in label_file.readlines()]


def get_message(result, threshold, top_k):
    if result:
        return 'Detecting:\n %s' % '\n'.join(result)

    return 'Nothing detected when threshold=%.2f, top_k=%d' % (threshold, top_k)


def process(result, labels, tensor_name, threshold, top_k):
    """Processes inference result and returns labels sorted by confidence."""
    # MobileNet based classification model returns one result vector.
    assert len(result.tensors) == 1
    tensor = result.tensors[tensor_name]
    probs, shape = tensor.data, tensor.shape
    assert shape.depth == len(labels)
    pairs = [pair for pair in enumerate(probs) if pair[1] > threshold]
    pairs = sorted(pairs, key=lambda pair: pair[1], reverse=True)
    pairs = pairs[0:top_k]
    return [' %s (%.2f)' % (labels[index], prob) for index, prob in pairs]


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--model_path', required=True,
        help='Path to converted model file that can run on VisionKit.')
    parser.add_argument('--label_path', required=True,
        help='Path to label file that corresponds to the model.')
    args = parser.parse_args()

    #hard code input parameters (these used to be command line parameters)
    input_height = 160
    input_width = 160
    input_depth = 3
    input_mean = 128.0
    input_std = 128.0
    num_frames = 100 
    output_layer = 'final_result'
    threshold = 0.1
    top_k = 3
    show_fps = False

    #use mobilenet image classifier
    model = inference.ModelDescriptor(
        name='mobilenet_based_classifier',
        input_shape=(1, input_height, input_width, input_depth),
        input_normalizer=(input_mean, input_std),
        compute_graph=utils.load_compute_graph(args.model_path))
    #labels are open and closed
    labels = read_labels(args.label_path)
    prob_arr = []
    with PiCamera(sensor_mode=4, resolution=(1640, 1232), framerate=30) as camera:

        with inference.CameraInference(model) as camera_inference:
            for result in camera_inference.run(num_frames):
                processed_result = process(result, labels, output_layer,
                                           threshold, top_k)
                message = get_message(processed_result, threshold, top_k)
                if show_fps:
                    message += '\nWith %.1f FPS.' % camera_inference.rate
                prob_arr.append(message)

            #get a screenshot of the garage    
            camera.capture('garage_state.jpg')
            #email the current state of the garage and a screenshot
            last_meas = prob_arr[-1]
            str_split = last_meas.split(" (") #splits status and probability
            status_split = str_split[0].split("\n ") #splits Detecting: \n open string
            email_subject = "Garage door is" + status_split[1] + " (Probability: " + str_split[1]
            sendMail('garage_state.jpg',email_subject)
            #get twilio info
            account_sid = os.environ['ID']
            auth_token = os.environ['TOKEN']
            
            client = Client(account_sid, auth_token)

            client.api.account.messages.create(
                to="xxxxxxx",
                from_="xxxxxxxx",
                body=email_subject)
            client.api.account.messages.create(
                to="xxxxxxxxx",
                from_="xxxxxxxx",
                body=email_subject)

if __name__ == '__main__':
    main()

Credits

Justin Lutz

23 projects • 38 followers

Quality manager by day, tinkerer by night. Avid runner. You can tell I'm a dad because of my jokes.

Don't let the mice in, close the garage door!

Things used in this project

Hardware components

Software apps and online services

Story

Schematics

Google AIY Vision Kit

Code

Garage Image Classifier Python code

Credits

Justin Lutz

Comments

Embed the widget on your own site

Don't let the mice in, close the garage door!

Don't let the mice in, close the garage door!

Things used in this project

Hardware components

Software apps and online services

Story

Schematics

Google AIY Vision Kit

Code

Garage Image Classifier Python code

Credits

Justin Lutz

Comments

Related channels and tags