Intelligent Object Detection

Introduction

This project is implementation of the idea that I submitted during ideas submission phase to enhance door security current process.

Old Security Monitoring ways needs security teams attention. They set down in-front of many screens to monitor every place in the building. This scenario is costly and not efficient. And if it can be handled by big organization, this will be hard in case of children forest camp for example, or someone wants to monitor his home.

One implementation of above idea can be to have walabot detect movement on camp boarder, in case of movement detection, walabot will alert raspberry which is connected to camera. Camera capture the picture. This picture will be analyzed by Google ML functions to capture a lot of info. For example

• label objects in picture. (persons, animals,..)

• Read text in pictures (Car number for example)

• Match persons against thieves' images database

What is different is Walabot can detect movement from inside as it can see through walls. This is more secure & save our hardware.

Imagine scenarios such detect if wild animal is moving toward children forest camp. or imagine if you can spot that children is moved out from the camp to the dark forest.

This can be implemented by using walabot that will trigger raspberry which is connected to camera. Raspberry will use firebase ML kit to label objects inside the picture and if labeled object is interesting, raspberry will use firebase to communicate with all security guards phones and send picture to them

As a proof of concept I make a small change on the above idea to use phone camera instead of separate camera. So as below architecture, raspberry will tell firebase that an object is very near. firebase will tell the connected phone, which will take picture for the object and use firebase ML SDK to label objects in the picture and to read any text in the picture,

Software

• First step is to create firebase project using firebase console then create needed hierarchy in real-time database

• Then use Android studio to create new android project

• Follow those steps to connect your application to firebase project.

• Above steps will link android project with created firebase project in previous step. However our application give user the ability to choose his firebase database on the fly.

• if user didn't fill the 4 info and click on "FIREBASE SIGN IN", he will be connected to the configured firebase in development time.

• To get those info return to firebase console and follow steps as bellow. Click on "Add app" Then click on "google-services.json"

• In android application, when click on "FIREBASE SIGN IN", below code will be executed

private void attemptLogin() {
    String apiKey = m_apiKeyView.getText().toString(); 
    apiKey = (apiKey.isEmpty()) ? "........" : apiKey ;
    String clientID = m_clientIDView.getText().toString(); 
    clientID = (clientID.isEmpty()) ? "......" : clientID ;
    String databaseURL = m_databaseURLView.getText().toString(); 
    databaseURL = (databaseURL.isEmpty()) ? "....." : databaseURL ;
    String storageBucket = m_storageBucketView.getText().toString();
    storageBucket = (storageBucket.isEmpty()) ? "....." : storageBucket;
    FirebaseOptions options = new FirebaseOptions.Builder()
            .setApplicationId(clientID) // Required for Analytics.
            .setApiKey(apiKey) // Required for Auth.
            .setDatabaseUrl(databaseURL) // Required for RTDB.
            .build();
    if (!FirebaseApp.getApps(this).isEmpty()) {
        FirebaseApp.initializeApp(this /* Context */, options, "secondary");
    }
    Intent FWrapperIntent = new Intent(this, FirebaseWrapperActivity.class);
    startActivity(FWrapperIntent);
}

• To read more about connect one android app to more than one firebase app check this link

• Application should listen to changes in the firebase realtime database

secondaryFirebaseApp = FirebaseApp.getInstance("secondary");
FirebaseDatabase database = FirebaseDatabase.getInstance(secondaryFirebaseApp);
DatabaseReference objectDetectedRef = database.getReference("applications/walabot/timestamp").getRef();
// Read from the database
objectDetectedRef.addValueEventListener(new ValueEventListener() {
    @Override
    public void onDataChange(DataSnapshot dataSnapshot) {
        // This method is called once with the initial value and again
        // whenever data at this location is updated.
        //Integer timestamp = dataSnapshot.getValue(Integer.class);
        //Log.d(TAG, "Value is: " + timestamp);
        //Take a picture automatically
        takePictureForDetectedObject();
    }
    @Override
    public void onCancelled(DatabaseError error) {
        // Failed to read value
        Log.w(TAG, "Failed to read value.", error.toException());
    }
});

• In above function when change is Realtime database happen, android device will get notified and function "takePictureForDetectedObject()" will be called to take picture

• To use the Google Cloud vision API, you need first to enable this API in your project

• Then picture will be sent to fiebase ML service to label objects inside the picture

FirebaseVisionCloudLabelDetector labelDetector = FirebaseVision.getInstance(secondaryFirebaseApp).getVisionCloudLabelDetector(options1);
Task<List<FirebaseVisionCloudLabel>> result1 =
   labelDetector.detectInImage(image)
     .addOnSuccessListener(
            new OnSuccessListener<List<FirebaseVisionCloudLabel>>() {
                 @Override
                   public void onSuccess(List<FirebaseVisionCloudLabel> labels) {
                        for (FirebaseVisionCloudLabel label: labels) {
                          String text = label.getLabel();
                          String entityId = label.getEntityId();
                          float confidence = label.getConfidence();
                        }
                   }
            })

• FIrebase ML also used to read text in image

FirebaseVisionCloudTextDetector textDetector = FirebaseVision.getInstance(secondaryFirebaseApp).getVisionCloudTextDetector(options1);
textDetector.detectInImage(image).addOnSuccessListener(
new OnSuccessListener<FirebaseVisionCloudText>() {
    @Override
    public void onSuccess(FirebaseVisionCloudText texts) {
        textView.setText(texts.getText());
    }
})

• You can monitor usage of Cloud Vision API from GCP concole

• In Manifest you should to add needed permissions (access camera, access internet,..)

<uses-feature
    android:name="android.hardware.camera"
    android:required="true" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.hardware.camera.autofocus" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />

• In below video you will see a demo how to use firebase realtime DB variable that when changed will trigger the android app to take a picture and send this picture to Google Cloud Vision API and get all labels in the picture

• Cloud vision required a certain image quality, please check this link. As taken pictures by mobile camera is not always produce quality images, maybe you will find application not respond on all taken pictures. So you need to try many times before you get a quality that can be processed by Cloud Vision API

• You can find useful info about API pricing here, and tutorials how to use this API here

Hardware

• To install raspberrian you can follow below video

• To install virtual keyboard in case if you have mouse only

• To enable Wifi from shell you can follow steps here.

• You can find useful Linux commands here.

• After install the raspian and connect Raspberry to wifi now you can setup Walabot SDK. To do this from shell follow below steps

apt-get update
wget https://s3.eu-central-1.amazonaws.com/walabot/WalabotInstaller/Latest/walabot_maker_1.0.34_raspberry_arm32.deb
mv walabot_maker_1.0.34_raspberry_arm32.deb walabotSDK_RasbPi.deb
sudo dpkg -i walabotSDK_RasbPi.deb 
pip install WalabotAPI --no-index --find-links="/usr/share/walabot/python/"

• Till this step I was using Raspberry pi zero. But I got notification from walabot team that walabot is compatible with Raspberry pi 3 and not with Raspberry pi zero, What you need is only to move the SD card from Raspberry pi zero to Raspberry pi 3.

• To setup firebase SDK on raspberry

sudo pip install requests==1.1.0
sudo pip install python-firebase
sudo pip install requests --upgrade

• Before using your walabot check this link as there is some tricks you need to takecare from based on your walabot type. This tricks related to powering the walabot

• Then you can apply some modification on "HelloWalabot.py" so that when distance between object and walabot less than 25 cm, raspberry update timestamp in firebase realtime database.

to update file we can use below commands

touch HelloWalabot.py
nano HelloWalabot.py

• File after update will be as follow

from __future__ import print_function
from sys import platform
from os import system
from firebase import firebase
firebase = firebase.FirebaseApplication('https://xxxx.firebaseio.com/', None)
import time
import WalabotAPI as wlbt
wlbt.Init()  # load the WalabotSDK to the Python wrapper
wlbt.Initialize()  # set the path to the essetial database files
wlbt.ConnectAny()  # establishes communication with the Walabot
wlbt.SetProfile(wlbt.PROF_TRACKER)  # set scan profile out of the possibilities
wlbt.SetDynamicImageFilter(wlbt.FILTER_TYPE_MTI)  # specify filter to use
wlbt.Start()  # starts Walabot in preparation for scanning
while True:
   wlbt.Trigger()  # initiates a scan and records signals
   targets = wlbt.GetTrackerTargets() # provides a list of identified targetsc
   
   for i, t in enumerate(targets):
       if(t.zPosCm > 25)
           ts = time.time() 
           firebase.put('applications/walabot/','timestamp',ts)  
           print(ts);
           time.sleep(120) # wait for 2 minutes 
wlbt.Stop()  # stops Walabot when finished scanning
wlbt.Disconnect()  # stops communication with Walabot

• To run the python file

python HelloWalabot.py

• Below video show you how to use Walabot to detect motion, if detected, raspberry will be alerted. Then raspberry to change timestamp value in firebase realtime database, which will notify mobile app to take the a picture and communicate with Cloud Vision API