Intelligent Closed-Circuit TV with Azure and Nvidia Jetson

Overview

In this article, we will describe the process of creating an Intelligent Closed-Circuit Television System with Microsoft Azure services and Nvidia Jetson hardware. To accomplish this, we will employ a set of Foscam IP cameras that are commonly used for home security and property surveillance. We will look at connecting our camera feeds up to a Virtual Machine in Microsoft Azure for storing our recordings offsite, then we will show how we can interact with those feeds to detect objects in near real-time through an Nvidia Jetson Nano device using YOLOv3-tiny object detection with Darknet inside an Azure IoT Edge Module. We will then push the detection results into Azure Time Series Insights for analysis.

Introduction

The Internet of Things allows us to make ordinary every-day items like thermostats and sprinkler systems smarter by simply giving them a line of communication to and from the internet. If you can put a sensor on a thing and make use of data surfaced up through the internet, you can find ways to improve almost anything!

This trend of IoT solutions is evolving though and becoming more than just smart sensors pushing readings and reacting to messages sent over the network. Small form-factor devices are now capable of performing highly intensive computations onboard without the need for offsite processing. This ability to function without the "I" in "IoT", is often referred to as "edge" computing and is commonly paired with Artificial Intelligence and Machine Learning workloads to produce intelligent insights at the IoT device site.

Microsoft has created the open-source Azure IoT Edge project to facilitate the development of IoT Solutions that operate in these "edge" environments where internet connectivity may be intermittent. The runtime allows you to develop IoT solutions that can be updated and deployed from the cloud, down to your device, where they can run with or without the presence of internet. This is accomplished by packaging workloads up as containerized "modules" which can be configured to run on an IoT Edge capable device. The runtime currently supports Windows/Linux x64 and Linux ARM32/64 environments. That means you can target pretty much anything from large scale N-Series GPU capable machines in Microsoft Azure down to small form-factor Raspberry Pi devices.

Hardware manufacturers are also positioning themselves for this next wave of IoT solutions by creating small form factor devices that are designed to run computationally intense workloads. Nvidia produces a variety of devices suited for these types of IoT solutions in their Jetson line of device offerings which contain onboard Graphical Processing Unit (GPUs). These include the beefy 512-Core Jetson Xavier, mid-range 256-Core Jetson TX2, and the entry-level $99 128-Core Jetson Nano.

In the remainder of this article, we will demonstrate how we can build a solution using IoT Edge to target an Nvidia Jetson Nano device to produce an intelligent IoT solution for monitoring Closed Circuit Television feeds. The system will allow for offsite archival of recorded video in the cloud and publish an event stream of objects detected in the camera feed to an Azure Time Series Insights instance. We will use this data to track how the system performs and offer up some ideas to enhance and improve the overall solution.

Hardware

The following items are required in order to successfully replicate this content:

• 1x RTSP IP camera like the Foscam FI9821P (indoor) or FI9828P (outdoor)
• 1x Nvidia Jetson Nano Developer Kit + a cooling fan (Cooling is very important!)

Note: Technically an RTSP capable IP camera is optional as our IoT Edge module will also work with a USB camera or YouTube videos.

Software

• Visual Studio Code
• 1x Azure Subscription (Free Trial available)

(Optional) Publish IP Camera Recordings to Microsoft Azure Linux VM

The first thing we need to do is ensure that our camera system can properly save recordings in the cloud. We will do this in a way that allows us to regularly backup recordings for up to a month, giving ample time for recovery of footage where necessary. If you are not using an RTSP capable IP camera, you can skip this section.

We will start by deploying an Azure Linux VM. Technically, it does not matter what distro you choose, as long as you now how to install the following packages. We will assume you are on a Debian based distro like Ubuntu in the examples.

1.) Create and ssh into a new Azure Linux VM instance

2.) Install vsftpd:

sudo apt-get install vsftpd  

3.) Configure vsftpd for local user access and PASV mode:

sudo nano /etc/vsftpd.conf   

Ensure the following options are set to disable anonymous logins allow local user access / write priveleges

# Allow anonymous FTP? (Disabled by default).  
anonymous_enable=NO  
#  
# Uncomment this to allow local users to log in.  
local_enable=YES  
#  
# Uncomment this to enable any form of FTP write command.  
write_enable=YES  

Next, add the following to the very top of /etc/vsftpd.conf being sure to replace the place holder values <>

listen_port=<enter listen port here>  
pasv_enable=YES  
pasv_max_port=<enter max passive port number here>  
pasv_min_port=<enter min passive port number here>  
pasv_addr_resolve=YES  
pasv_address=<yourVMDomain.cloudapp.net>  

The listen_port should be a port number that is NOT in use by another service, similarly the range of ports used for psv_max_port and pasv_min_port should not be in use. It is advised to use a range of no more than 10 ports for psv_max_port and pasv_min_port. This will allow you to reserve ports for other services but will limit your max ftp connections to 10, adjust according to your use case.

The full DNS for yourVMDomain.cloudapp.net can be obtained in the Azure portal in the Overview section for your Virtual Machine. If you have configured a custom DNS that resolves to your VM, you may use that value here as well.

Once configured, restart the service:

sudo service vsftpd restart  

4.) Allow traffic to/from the FTP service in the Azure Network Security Group

In the Microsoft Azure Portal, locate the Network Security Group for your VM and create a new rule to allow traffic from the listen_port configured earlier. For example, if you chose port 7001, it would look like this:

Next, do the same for the psv_max_port and pasv_min_port range. For example, if you chose pasv_min_port = 10050 and psv_max_port = 10060

5.) Configure cameras to publish recordings to Azure VM over FTP

Note: Depending on your chosen camera, the available options may be different.

We will start by logging into our Foscam camera, selecting "Settings" => "Network" => "FTP Settings"

Fill in the appropriate values and select Test, note that the Camera directory is specified in the FTP server connection. This is the location where camera recordings will be stored to.

If everything is configured properly, you should receive a notification that the test was successful

6.) Setup a cron job to remove video files older than 8 days

Run the following:

crontab -e  

And add the following line for the camera in question:

@daily find /home/myuser/Camera/* -type f -mtime +8 -exec rm {} \;   

7.) Configure Scheduled Backups for Linux VM in Azure

Navigate to the Overview for your VM and select "Backups", this will present you with options for configuring scheduled backups of your VM. I like to keep 4 weekly backups at any given time allowing me to keep around a month of video files available for recovery in addition to everything else that is stored on the server. Your use case may vary, but this works great for me and allows me to sleep very comfortably at night =)

An example backup policy configuration is provided below:

Adding Object Detection to our Camera Feed(s)

We previously setup our camera feeds to record into Microsoft Azure using a Backup policy to ensure that past recordings are available for approximately 1 month. Next, we are going to use an Nvidia Jetson Nano device to augment our camera system by employing YOLOv3-tiny object detection with Darknet inside an Azure IoT Edge Module.

We will configure our device by following the instructions available @ https://github.com/toolboc/IntelligentEdgeHOL.This GitHub repository contains full instructions on how to setup and deploy the YoloModule to an Nvidia Jetson Nano device. If you do not have an available IP camera, you can still follow along with the content by connecting up a compatible USB web cam OR by using a YouTube video link.

If you are using a supported Foscam camera, you will want to ensure that you are able to connect to it's RTSP stream. Instructions for finding this value can be obtained in this forum post. Once you know the appropriate value, it can be provided in the steps for Configuring the YoloModule Video Source.

With that said, go ahead an follow through the README provided @ https://github.com/toolboc/IntelligentEdgeHOL and return to this article once you have completed the steps and verified that object detection is working on your camera stream.

Once you get your stream up and running, you should be able start detecting the 80 supported objects at around 10 frames per second, not too bad for a $100 device!

Pushing Detected Object Data into Azure Time Series Insights

Now that we have the ability to detect objects in our camera feed, let's push it into a service that will allow us to easily dig through the data to perform analysis on our results. Azure Time Series Insights is built to store, visualize, and query large amounts of time series data, such as that generated by IoT devices. If you want to store, manage, query, or visualize time series data in the cloud, Time Series Insights is likely right for you.

This will allow us to extract some rudimentary insights that may allow us to build something even more interesting. For example, imagine getting an alert when the mail truck is actually at the driveway, counting wildlife species using camera feeds from the National Park Service, or being able to tell that people are in a place that they should not be and counting them over time!

Start by heading over to Azure and navigate to the resource group that you created in the previous step. This resource group should contain an IoT Hub instance. Add a new Time Series Insights environment into the Resource Group and select an appropriate tier for deployment.

Next, configure the Event Source to point to the IoT Hub you created in the previous step and be sure to create a new IoT Hub Consumer Group for it.

Once the instance has finished deploying, you can head on over to your Time Series Insights Portal where you can begin working with your detected object data.

You will notice that my camera feed produces zero results during evening hours. I learned that YOLOv3 object detection just wouldn't detect anything when there isn't enough light outside. Also, the detection algorithm always thinks that my girlfriend's car is a train. This could be remedied by retraining the YOLO model to detect exactly what we want instead of using it's pre-trained model. Even with these issues, some important data can be extracted.

Looking at the model, I can tell exactly when the "train" is coming and going. Similarly I can notice things like when a person was detected and can reference my video recordings in Azure to verify that there is in fact a person in view at the recorded time.

For example, in this view it seems pretty likely that a person or persons were in view of the camera at around 12:35 PM on 7/16

Let's roll the tape!

Similarly, if we adjust our query to a bit wider range, we can see that it looks like 3 cars have passed by the driveway in the last hour. I imagine this type of data could be very useful to parking garages or city planners.

Conclusion

We have demonstrated how to setup a Closed Circuit Television System by using a Virtual Machine in Microsoft Azure for storing recordings offsite. We then enhanced that system by employing a relatively inexpensive Nvidia Jetson Nano device to analyze frames on the camera streams to detect objects in the feed and publish them to an Azure IoT Hub by running a special Azure IoT Edge module on the device. This allowed us to use GPU accleration to perform detection of 80 unique objects at around 10 frames per second. Finally, we pushed that data into an Azure Time Series Insights instance to perform analysis on our data.

It is understandable that this content may not be specific to your use case, but it is my hope that it helps to enable the creation of similar IoT solutions intended for deployment in "edge" environments. We have identified some issues with the object detection algorithm, but it is important to note that it can always be retrained or replaced with something more suitable. Once we have an AI service in place that is able to accurately detect what we are interested in, we have shown that you can very easily extract insights by leveraging a service like Time Series Insights to quickly drill into collected data points. As such, we have solved quite a few large general problems that should aid in developing something more specific.

If you enjoyed this content and want to stay up to date on my latest projects, you can follow me @pjdecarlo on Twitter.

I am always interested to learn how others interpret this content for their own use. If you have any cool ideas to enhance this project or ideas of your own that were inspired by this content please share them in the comments.

Until next time, happy hacking!