Alexa Doorman: Who Is at My Door?

0. Published Skills

Smart Home Skill: https://www.amazon.com/dp/B0792RP9F2/

Custom Skill: https://www.amazon.com/dp/B07935KVCH/

1. About

Be notified of when people (or things) are at your door! Simply ask Alexa to check who's at the door. In this guide, you will be building a smart camera, an object detection API, a Alexa Smart Home Skill, and a Custom Skill! Your smart camera will also be able to capture activity when motion is detected.

Note: When notifications are publicity available from Amazon Alexa, then that feature will be implemented. For now, SMS :)

2. Motivation

I make a lot of Alexa skills, but this will be my first smart home skill! It's both a challenge and an interesting task to take on. I'm a software engineer by trade but tinkering with hardware has always held my interest. Hopefully, this guide will help both hardware and software folks who want to get into either side.

3. Hardware Setup

Choose one of the three available setups. You can even roll your own hardware as long as it has the capability to stream video/send images. The Python client will work anywhere where Python is installed (with OpenCV).

All the below require a compute device. You can run the computations on your computer, Raspberry Pi (if you want to use it only as the compute), or any cloud hosted compute.

3.0 /dev/video0

Sometimes you will need to run sudo modprobe bcm2835-v4l2 to ensure that your webcam is mounted to /dev/video0 if you are on Linux. Without it, OpenCV video capture will fail!

3.1 Using ArduCam, Arduino, + Compute

The ArduCam/Arduino will be responsible for sensor data, which in our case is the camera capture. You will need a computational server like Raspberry Pi or Intel Edison or you can use a ESP8266 WiFi chip to send requests to the API server. It all depends on what you have on hand.

In this setup, you will need a Arduino device, a compatible camera such as the ArduCam, and a computational device such as Raspberry Pi or you can use your desktop/laptop. This guide will use a Raspberry Pi.

Setup your ArduCam according to this PCB design, here's the circuits.io link:

https://circuits.io/circuits/5729301-arduino-uno-r3-arducam-mini-2mp

Connect the Arduino to the Raspberry Pi through USB and also connect the Arduino's barrel jack to a power supply.

Obtain https://github.com/alexa-doorman/Arduino and add it to your Arduino IDE libraries.

Flash ArduCAM_Mini_LowPowerMode.ino located under Examples > ArduCam > mini.

3.2 Using Intel Edison with Arduino Breakout and UVC-compatible camera

3.2.1 Setting up jubilinux

Setup jubilin1ux to boot from an SD card. eMMC does not have enough space for anything besides the streaming client. Download the jubilinux image and use this tutorial: https://sarweshcr.blogspot.com/2015/11/Boot-Intel-Edison-from-SD-card-with-Debian-or-Ubilinux.html

You may end up having problems with bring up your WiFi. You need to edit /etc/avahi/avahi-daemon.conf and find and set this line like this: disallow-other-stacks=yes. You can then use the script I provided below to setup WiFi. I also had problems with not being able to use sudo. I ran these lines to fix:

(source)

$ chmod 1775 /
This will correct the permissions for your / directory
$ sync
This will make sure the change is written to the filesystem

Intel has dropped support for the Edison so it's no longer fruitful to use the stock image. I like jubilinux because it ships with Python 3.5 so you won't need to compile Python 3. To install jubilinux, visit the official site. It the same setup as flashing other images using flashall .

If the device does not flash rootfs then consider booting with Ubuntu 14.04 32-bit to flash the device. You will need to setup dfu-util to flash on Ubuntu.

Once jubilinux has been flashed, confirm that you have Python 3 installed by running python3 -V .

After you confirm a successful flash, complete these steps:

• Edit /etc/rc.local to set USB host mode for the OTG port
• Setup Wi-Fi (you can use this script to setup Wi-Fi; simply copy paste into bash)
• Mount the micro SD card

3.2.2 Setting up UVC-compatible camera

Ensure that the toggle switch near the micro-USB ports is flipped TOWARDS the USB OTG port.

If this is your first time setting up a camera in Linux, consider following this guide.

To verify the camera has mounted, run ls -l /dev/video1

3.3 Using Raspberry Pi and Raspberry Pi Camera

Install the Raspberry Pi camera in the camera port that's supports the ribbon connection. The port also says CAMERA . Refer to this guide for full installation.

To verify the installation:

pi@device $ vcgencmd get_camera
supported=1 detected=1

That's it. OpenCV will detect it on /dev/video0.

I strongly suggest using Docker if you're using this setup for the Software section.

I've tested on Debian Stretch (which is the latest Raspbian release).

4. Software Setup

4.0 Architecture Diagram

High level common requirements for the streaming client and the detection API:

• Python 3
• OpenCV 3
• Flask

For the detection API:

• All of the requirements from the streaming client
• Tensorflow
• Cython
• Darkflow
• Redis

Please use the provided requirements.txt with pip where possible.

4.0 Ports

Make sure that ports 5001 and 5000 are open. I like to use ngrok to test my Flask servers.

4.1 Prerequisites

4.1.1 Installing Python 3

Windows, macOS

I strongly suggest using the Anaconda3 distribution of Python. Most of the binaries are pre-compiled. You can skip straight to cloning and running.

Linux (debian)

Sometimes Python 3 is installed by default. I've tested with at least Python 3.4. Please ensure you have at least that version but feel free to try lower versions.

Otherwise, You will need to compile Python 3 from source OR use pyenv-installer.

4.1.2 gstreamer + gi (Python 3 bindings)

4.1.2.a gstreamer

Windows / macOS / Linux

Install gstreamer from the official site: https://gstreamer.freedesktop.org/download/

4.1.2.b Python bindings

Windows

Install the Python 3.4 bindings from the pygi-aio-3.14.0_rev22-setup.exe executable here: https://sourceforge.net/projects/pygobjectwin32/files/

You only need the gstreamer tools.

To test your installation, run these commands:

$ python3
>>> import gi
>>> gi.require_version('Gst', '1.0')
>>> gi.require_version('GstRtspServer', '1.0') 

Linux (stretch)

If you're not on stretch, you need to compile gstreamer, gst-rtsp-server, gst-python, and gst-plugins-ugly (with --enable-introspection=yes during ./configure).

$ sudo apt-get -y install gstreamer1.0-rtsp gir1.2-gst-rtsp-server-1.0 python3-gst-1.0 gstreamer1.0-plugins-ugly 

To test your installation, run these commands:

$ python3
>>> import gi
>>> gi.require_version('Gst', '1.0')
>>> gi.require_version('GstRtspServer', '1.0') 

4.2 Installing YOLOv2 Object Detection API (on compute device)

IMPORTANT: You will need to download the model files before doing anything. Download them here (or from the attachments) and place the model directory in the project folder once you done. If you are running on low memory devices (like the Raspberry Pi or Intel Edison, download the tiny-yolo model instead!)

4.2.0 Docker Images

NOTE: If you can afford to use Compose, I've provided the necessary files for both the GPU and CPU versions to run this API. Simply run docker-compose up --build and you can skip onto the next step.

4.2.1 Raspberry Pi

~ $ git clone 
~ $ sudo docker run --name redis-yolo -d redis
~ $ sudo docker run --link redis-yolo:redis -e REDIS_URL=redis://redis:6379/0 --volume "/home/pi/yolo-detection-api:/src/app" -p 5001:5001 -d doorman/yoloapi:rpi

That's it! Check http://<ip-of-device>:5001 or try to post to http://<ip-of-device>:5001/detect

I recommend using the tiny-yolo-voc version both attached to this post and also located here: https://drive.google.com/drive/folders/1NYtW4w2EjasFzvNQt_J6jduWeNWUIxyQ.

{
   "results": [
       {
           "bottomright": {
               "x": 588,
               "y": 539
           },
           "confidence": 0.5031049251556396,
           "label": "person",
           "topleft": {
               "x": 136,
               "y": 35
           }
       }
   ],
   "status": "success"
}

4.2.1 Other Architectures

Use the appropriate doorman/yoloapi:* image tag.

Try to use the full size model where possible,

• armhf
• arm32v7
• 64
• 64-gpu

4.2.2 The Hard (but sometimes only) Way

We are going to be installing darkflow which requires some pre-requisites to be installed.

If you cannot install the following packages you will have to compile them depending on what device you are running this on. Either use pip or conda

(env) $ pip install cython
(env) $ pip install numpy
(env) $ pip install tensorflow

If tensorflow is not found, you need to compile it. On Raspberry Pi, you can use these installation notes to get it easily if you are on Python 2.7/3.4 (which ships with the Pi).

Tip: If you can install/compile Tensorflow with GPU support, you should.

You will also need OpenCV 3 installed. You can install it using Anaconda (conda).

The other option is to compile it from source. A great resource for doing just that: https://www.pyimagesearch.com/2017/09/04/raspbian-stretch-install-opencv-3-python-on-your-raspberry-pi/

Note: Ensure that you have Redis running. This guide is decent to set it up as a service.

(env) $ git clone https://github.com/alexa-doorman/yolo-detection-api.git  
(env) $ cd yolo-detection-api 
(env) $ pip install -r requirements.txt 
(env) $ python api.py 

Note: You will need the IP of this Flask server for the streaming client (next section).

Make sure you get a 404 error page when you visit http://<ip-of-yolo-detection-api-server>:5001. If you get a 500 error, it's probably because you forgot to download the model folder.

Tip: To throttle requests, set the THROTTLE_SECONDS environment variable.

4.3 Installing Streaming Client (on client device, e.g. Intel Edison, Arduino)

Enable/source your Python 3 environment, ensure pip is installed.

(env) $ export REMOTE_DETECT_SERVER=<ip-of-yolo-detection-api-server>/detect 
(env) $ git clone https://github.com/alexa-doorman/stream-client.git  
(env) $ cd stream-client  
(env) $ pip -r  
(env) $ python app.py 

Visit http://<ip-of-streaming-device>:5000/live to ensure the webcam is functioning.

To test the detection API, visit http://<ip-of-streaming-device>:5000/stream-detect

The API should draw boxes around objects it detects.

Hint: You can figure out the IP of your device using ifconfig or ipconfig and looking for either the wireless device or the ethernet connection.

Take a deep breath. You've gone through a lot. The rest is easy as pie.

4.4 (Optional) Link Account/Authentication App + Custom Skill Endpoints

This app is in charge of interacting with the Alexa skill and authenticating users. It's going to be deployed with Zappa.

The app also hosts the Custom Skill endpoints which is implemented using flask-ask.

The server I've deployed is located here: https://carxxcdjwe.execute-api.us-east-1.amazonaws.com/dev

You will need to setup an app to use Login with Amazon (DOORMAN_LWA_KEY and DOORMAN_LWA_SECRET environment variables). Visit the official page and obtain the client ID and secret key. See the reference below:

For Zappa deployments, you will need to create a local virtual environment.

You need to ensure your AWS credentials are properly set. This guide will help you get setup.

Once you've activated your environment run these commands:

(env) $ git clone https://github.com/alexa-doorman/link-account-app.git
(env) $ cd link-account-app
(env) $ touch .env  # Check below to see how to format your keys
(env) $ pip install
(env) $ zappa deploy dev  # or if you want to customize use: zappa init

You should have gotten a AWS API Gateway link to view your application (e.g. https://carxxcdjwe.execute-api.us-east-1.amazonaws.com/dev). You will need to go in and edit all the hardcoded URLs in the templates and views with your URL if you can't setup a Custom Domain in the API Gateway Console.

.env

DOORMAN_LWA_KEY=amzn1.application-oa2-client....
DOORMAN_LWA_SECRET=
SECRET_KEY=
OA_CLIENT_ID=
OA_CLIENT_SECRET=
OA_REDIRECT_URIS=https://url1.com,https://url2.com
OA_SCOPES=profile
QUERY_CLIENT_ID=
QUERY_CLIENT_SECRET=
QUERY_REDIRECT_URIS=

The following environmental variables relate to the Custom Skill:

QUERY_CLIENT_ID=
QUERY_CLIENT_SECRET=
QUERY_REDIRECT_URIS=

The following relate to the Smart Home Skill:

OA_CLIENT_ID=
OA_CLIENT_SECRET=
OA_REDIRECT_URIS=https://url1.com,https://url2.com
OA_SCOPES=profile

After deploying, you should see the homepage:

Follow the prompts and if you are redirect to this page, then you have successfully setup Login with Amazon.

4.4 (Optional) Alexa Smart Home Skill

If you are at this step that means you have publicly reachable IP addresses for the client streaming Flask server and the YOLO object detection API. ngrok is a nice tool to test before setting up something more permanent. You can set static IPs to your devices but they can change depending on your ISP. I've also used No-IP and haven't encountered problems with it.

I've forked off the official examples Amazon provides for Smart Home Skills and modified it to obtain RTSP URI from our DynamoDB server. You can only view streams with the Smart Home Skill which is more of a limitation by Amazon than what I can do.

You first need to clone the repository located here then zip the contents of sample_lambda/python. Make sure the zip has the folder called lambda.py in the root!

Here is a link to the guide that Amazon Alexa has posted. You should go through it if you have never deployed a skill before.

Essentially, you want to upload that to AWS Lambda, grab the ARN and head to the Alexa dashboard and make a Smart Home Skill which points to that. Make sure to There are no sample utterances or any training to do. Since we've setup our Link App, we have access to all the information to obtain a user's RTSP server URI. Below is a screenshot of the configuration screen:

You will also need to link DynamoDB:

Here are example requests and response that you can follow to make sure that your skill works. Test them in the Lambda dashboard.

https://github.com/alexa-doorman/alexa-smarthome/tree/master/sample_messages/CameraStreamController

4.4.1 Examples Phrases

(Alexa,) show (my|the) {target} (camera|camera feed|feed)

(Alexa,) show (my|the) feed from (my|the) {target} (camera|camera feed|feed)

(Alexa,) hide (my|the) {target} (camera|camera feed|feed)

(Alexa,) hide (my|the) feed from (my|the) {target} (camera|camera feed|feed)

4.5 (Optional) Alexa Custom Skill

If you've deployed the Link Account app, then all the endpoints for Alexa are setup at http://<zappa-deployment.com>/alexa!

4.5.1 Interaction Assets

Intent Schema: https://gist.github.com/exp0nge/6441b159ec75ae3229dead56b4669186

Utterances: https://gist.github.com/exp0nge/b9a7544410503703f973c2f087165fc8

4.5.2 Configuration

5.5.3 Example Phrases

Alexa, ask Doorman to check my door

Alexa, ask Doorman what is at the door?

Alexa, ask Doorman to check my camera status

5.0 Demo

5.1 Video

5.1.1 Smart Home Skill

5.1.2 Custom Skill

5.2 Screenshots

5.2.1 Smart Home Skill

5.2.2 Custom Skill

6.0 Feedback/Issues/Questions

If you have any questions, feedback, or run into any issues, feel free to submit a issue on the respective repo on GitHub or comment here. I'll try my best to answer them!