Bird Detector Based on Microsoft Azure and Raspberry Pi

Introduction

Recently, I entered a cloud skill challenge named “Azure Data and AI Challenge”, which was initiated by Microsoft to enable developers to learn Azure Data & AI technologies as well as Accessibility in terms of a part of the journey of exploring the Diversity and Inclusion perspective. As far as I am concerned, I like this form of learning very much since every module includes clear purpose, detailed concept explanation, step-by-step tutorial, as well as free Azure cloud resources for users to practice. This reminds me of what I had experienced in the classroom. Not only should we know “what”, but also learn to know “how”.

One particular learning module I have come across is called “Classify endangered bird species with Custom Vision”. In this module, we use Azure Cognitive Services Custom Vision to create a model to identify species of birds! More specifically, we can build a machine learning model that will identify a bird in an image by species.

Coincidentally, a dove family joined us last week and built a nest in front of my window. What a surprise! They laid two eggs in the nest this morning. These little creatures have brought us endless happiness. My son always gets as close to the window as he can to watch them carefully. However, doves looked a bit scared when humans get close.

After completing the learning, an idea came to me. Why not to use Azure Custom Vision to identify the bird next to our window? However, if we follow the steps in the learning path provided above, we should upload the image and get the results on PC, which is quite inconvenience for real-time monitoring. After some thought, I decided to use Raspberry Pi and web camera as IoT terminal to achieve daily surveillance. Then, we do not have to show up next to the window to watch the birds. The PIR sensor will trig the process to take the image and the image will be transmitted to Azure Custom Vision to get the detection results. Then, the Raspberry Pi will upload the result image to the Azure Blob Storage, which is accessible from the Internet everywhere.

If you are interested in this project, you can learn the path that I created in this collection: collection for AI Bird. I have invited my friends to complete the learning modules on Twitter: Twitter Post. After complete these modules, you can hardly wait to make your own project based on Azure AI and IoT technologies.

System Overview

As shown in Fig. 2, the whole system consists of Raspberry Pi terminal, Azure Custom Vision, Azure Blob Storage and user device. We connect a USB web camera (or Raspberry Pi official Camera) and a PIR sensor to Raspberry Pi for image capture and bird detection. Once a bird gets close to PIR sensor, it will send a signal to Raspberry Pi indicating that some kind of animal is approaching. Then the Raspberry Pi will take a picture by the camera and send it to Azure Custom Vision service to get the results. We will use PIL library to draw a rectangle to highlight the object that we detect in the picture. Finally, the Raspberry Pi will upload the picture to the Azure Blob Storage for users to review anywhere.

Hardware

Microsoft LifeCam HD-3000

Raspberry Pi

PIR Sensor

HMDI Screen (Optional)

Azure Resources

We will use a Microsoft account in the project. If you don't have, please apply for one. All resources that we use in this project is provided as sandbox. The sandbox may only be used to complete training on Microsoft Learn. Use for any other reason is prohibited, and may result in permanent loss of access to the sandbox. Please note that all resources in your sandbox are available in 4 hours. They will be discarded once the time is up.

Project Instruction

Step 1 to Step 6 is quite similar to those in module “Classify endangered bird species with Custom Vision”. The difference is that we create an Object Detection project rather than Classification project here. If you just want to achieve bird’s classification, please refer to the steps provided in the learning module.

1. Download the dataset

Data is the first thing we need to create a machine learning model. We'll use a subset of the NABirds dataset from the Cornell Lab to train our model. Download the zip file that contains the dataset: bird-photos.zip

When the download is finished, unzip the file, as shown in Fig. 3. Note the folder location because you’ll need it in a later step. We will find that there are totally 16 species of birds in the fold. Each kind includes about 120 images.

2. Create Custom Vision API resources

Next, we'll create an API resource in Azure Cognitive Services Custom Vision.

In the Azure portal, select Create a resource.

Search for custom vision. In the Custom Vision card in the search results, select Create.

On the Basics tab, enter or select the required values:

⦁ Select your Azure subscription. If we use sandbox, the default subscription is Concierge Subscription.

⦁ The default resource group is learn-******

⦁ Enter a name for your new Custom Vision service resource (for example, BirdCustomVisionDemo).

⦁ Under Training Resource: Set Training location to the nearest point.

⦁ Set Training pricing tier to Free F0 (2 Transactions per second).

⦁ Under Prediction Resource:

⦁ Set Prediction location to the same one as we choose above.

⦁ Set Prediction pricing tier to Free F0 (2 Transactions per second).

⦁ Select Review + create.

⦁ Select Create.

Please refer to Fig. 4 for more details.

3. Upload the data

There are two ways to upload images. First, in Custom Vision portal, we can select, upload, and then tag images. Alternatively, we can use the Custom Vision SDK if we have a large amount of data, image classes, and tags to upload. Here we use Custom Vision portal for image uploading.

Go to https://www.customvision.ai/projects and sign in. Select New project.

In Create new project:

⦁ For Name, enter a project name of your choice.

⦁ For Description, enter a short description of the model. (for example, DetectBirds).

⦁ For Resource group, Select the resource group you created in the Azure portal.

⦁ For Project Types, select Object Detection.

⦁ For Domains, select General. See Fig. 5 for more details.

Select Create project.

Now, it’s time to add images and tags for a bird species in the pictures. In your Custom Vision project, select Add images.

In Open, go to the birds-photo folder where you extracted the images files from the dataset zip file.

Open a bird species folder.

Select Ctrl + A to select all the images in the species folder, and then select Open.

After uploading, we should tag the object in every picture. Please refer to Fig. 6 for more details.

Repeat the preceding step to upload the photos in each bird species folder in the downloaded dataset.

4. Train the data

We've created our dataset in Custom Vision. Now, it’s time to train our model. In the Custom Vision portal, select our project. In the top menu bar, select Train. In Choose Training Type, select Quick Training, and then select Train. It’s shown in Fig. 7 as follows.

When training finishes, information about how the model performed for the training iteration is presented in Figure 8.

Custom Vision displays three metrics when we test your model. Metrics are indicators that can help you understand how your model is performing. The indicators don't indicate how factual or accurate the model is. The indicators tell you only how the model performed on the data you provided. How well the model performed on known data gives you an idea of how the model will perform on new data.

Precision: If your model predicts a tag, this metric indicates how likely it is that the correct tag was predicted.

Recall: Of the tags the model should predict correctly, this metric indicates the percentage of tags your model did predict correctly.

Average Precision: Measures model performance by computing the precision and recall at different thresholds.

5. Test the model

Let's test our model and see how it performs on unseen data. We'll use an image of a bird from an internet search.

In the Custom Vision portal, select our project. In the top menu bar, select Quick Test. In Quick Test, paste the URL in Image URL and then press Enter to test the model's accuracy. The prediction is shown in the Fig. 9 as follows. We will find that the results including the object coordinates in the image, which is highlighted by red rectangle.

6. Deploy the model

We can deploy to an endpoint by getting the prediction URL or by consuming the API in an application.

In the Custom Vision portal top menu bar, select Performance. Select Publish. In Publish Model, select Prediction resource, and then select the name of the prediction for your Custom Vision project. Select Publish.

Select the Prediction URL tab. In How to use the prediction API, we can get the Key as shown in Fig. 10.

Furthermore, we shall get the Endpoint and the Project ID for further use. They can be accessed on the Settings page as shown in Figure 11.

7. Assemble the hardware

In this project, a Raspberry Pi, a USB web camera (the official Raspberry Pi camera is OK) and a PIR Sensor are necessary. If you have a HDMI Screen, you can see the results on the screen directly. If not, you can install VNC server on Raspberry Pi, and interface with it by VNC client on your PC or other mobile devices. Here are the modules I use:

⦁ Microsoft LifeCam HD-3000

⦁ Raspberry Pi 4

⦁ PIR Sensor.

⦁ Waveshare 7 inch HDMI LCD (B) (800*480)

The PIR Sensor is equipped with three pins, i.e. Vcc, GND and Out. Vcc and GND will be connected to the 5V and GND pin on Raspberry Pi, while Out pin will be connected to BCM pin NO. 4. The protype is shown in Figure 12.

8. Install necessary libraries

We will use Azure Custom Vision Python SDK, Azure Storage Blog Python SDK, and USB camera. Then we shall use pip tool and apt-get to install necessary components as follows.

pip3 install azure-cognitiveservices-vision-customvision
pip3 install azure-storage-blob
pip3 install msrest
sudo apt-get install fswebcam

To create an Azure storage account and see more information on how to use Azure Storage Blobs client library for Python, please refer to this docs: Azure Storage Blobs client library for Python.

The connect string can be reviewed on Azure Portal as shown in Fig. 13.

9. Download and run the code

You can download the python code from Github: https://github.com/shijiong/AzureCustomVisionBirdDemo. Please make sure that you input the prediction key, the prediction endpoint, project ID and publish iteration name of your Custom Vision project in line 14 to line 17. And the connection string, container name and blob name in line 19 to line 21 should be replaced with yours. Please refer to Fig. 14 for more details.

Secondly, in line 33, we specify the image resolution with 1280*720, which maybe different in your project. It’s quite important in that we will draw a rectangle on the image according to the results of the prediction API in line 46. If the resolution of the image is different, we should alter it or we will not get the right position of the recognized object on the image.

Thirdly, I set the threshold of the prediction probability to 0.9 in line 41, you can change it to meet your scenario.

Lastly, if you want to review the prediction image on Raspberry, you can uncomment the codes from line 49 to 52. However, the upload process will be blocked until you close the image.

I deploy the device next to my window. Copy the code to your Raspberry Pi, open it with Thonny, the preinstalled Python IDE on Raspberry Pi, and run. Once the PIR sensor is trigged, the image will be captured and transmitted to the Azure Custom Vision to get the prediction results. The recognized object will be highlighted by red rectangle as shown in Fig. 15.

And finally, the image will be transmitted to Azure Blob Storage as shown in Fig. 16. You can review it from anywhere you are.

Summary

In this tutorial, we have designed a bird detector based on Azure Custom Vision, Azure Blob Storage and Raspberry Pi. And then give the demonstrations of dataset uploading, data training, model deployment, hardware building, and operation procedure. Hope this will be useful for those who need Azure Custom Vision Services on Raspberry Pi device.