Microsoft Azure Machine Learning and Face Detection in IoT

Applications of Future draw their intelligence from a wide variety of source (Web 3.0 connected databases). They would learn continuously and be aware of the ecosystem they work in. Machine Learning is our first step on that path.

Machine Learning is a field in Artificial Intelligence that deals with predictions. It is been in use in various forms for quiet some time. For example, the suggestions we get when we go online to buy something is powered by ML. The Facebook’s friend suggestions too!

The process of Machine Learning is also a continuous process where we collect data and apply various known and established algorithms that generates scores and then we pick the best model and make is available for consumption. Predictions in ML get stronger and smarter and precise as more and more data feeds in.

Microsoft Azure and Machine Learning

Microsoft Azure Machine Learning is a powerful cloud based service for predictive analytics. It enables you to define models and run your experiments in the cloud to analyze data and find patterns.

Microsoft Azure Machine Learning Studio is a tool that gives a drag-drop interface to create models and try them quickly. You can share your models with the community to use or use what others have shared through Cortana Intelligence Gallery.

Microsoft packaged their Speech and Face Recognition ML models into Microsoft Cognitive Services (Project Oxford). These are not just recognitions but the APIs can pull a lot more context from the data and thus the applications we build with these can be more productive and personal.

We will be exploring Face API in this experiment. First we will understand what the service can provide and then see a use-case.

Microsoft Face API,

It is a part of Microsoft Cognitive Services, a powerful cloud service with face detection and face recognition functions.

• Face Detection means to find a face region in an image. It is represented by a rectangle. The API also has functionality to find various features/attributes of the face – age, gender, pose, glasses and facial hair. There are many online applications which use Face API in the background.

• Face Verification is ability to find similarity between two faces. This API can compare two face, find all similar faces and like.

To understand how to use the service, we need to first understand a little about the data organization. The object hierarchy is as follows –

There are Person Groups which can have up to 1000 Person in it. Each Person can have up to 248 Face associated. An image with a face can be searched in a people group, a person or against another face.

You can have a Person Group of Family members, another of Friends and like. Your Family person group can have Person object and each represent a family person. And then you can have multiple face images associated with each Person. It works well with face and near face images.

So before we start, activate your free subscription for Face API in Microsoft Azure Cognitive Services. And here is API documentation for reference.

Let’s go the Top-Down approach!

Person Groups

We shall first create a Person Group. PersonGroup is a logical grouping of Persons. You can create one universal set or you can create multiple person groups depending on the use-case. The limit is 1000 Person objects in one PersonGroup.

public class PersonGroups
{
    public string personGroupId { get; set; }
    public string name { get; set; }
    public string userData { get; set; }
}

PersonGroup has an ID which we define and we provide a ‘name’. We can also associate some ‘userData’ with it. For example a description or may be something internal to our use.

public static async Task<string> AddPersonGroups(string personGroupId, string name, string userData)
{
    string uri = HttpHandler.BaseUri + "/" + personGroupId;
    string jsonString = "{\"name\":\"" + name + "\", \"userData\":\"" + userData + "\"}";
    HttpContent content = new StringContent(jsonString, Encoding.UTF8, "application/json");
    
    HttpResponseMessage response = await HttpHandler.client.PutAsync(uri, content);
    response.EnsureSuccessStatusCode();
    
    string responseBody = await response.Content.ReadAsStringAsync();
    return responseBody;
}
BaseUri: https://api.projectoxford.ai/face/v1.0/persongroups

Persons

A person object represents a person in the person group. When we create a new person object, we need to associate it with a personGroupId.

class Persons
{
    public string personId { get; set; }
    public string name { get; set; }
    public string userData { get; set; }
    public List persistedFaceIds { get; set; }
}

We need to associate a ‘name’ and can associate ‘userData’. A personId is returned when a new person is created, but it has no Face associated.

public static async Task<string> CreatePerson(string personGroupId, string name, string userData)
{
    string uri = HttpHandler.BaseUri + "/" + personGroupId + "/persons";
    string jsonString = "{\"name\":\"" + name + "\", \"userData\":\"" + userData + "\"}";
    HttpContent content = new StringContent(jsonString, Encoding.UTF8, "application/json");
    
    HttpResponseMessage response = await HttpHandler.client.PostAsync(uri, content);
    response.EnsureSuccessStatusCode();
    
    string responseBody = await response.Content.ReadAsStringAsync();
    return responseBody;
}

Faces

A face object is associated with a Person and personGroup. There can be as many as 248 Faces that can be associated with a person.

public class FaceData
{
    public string persistedFaceId { get; set; }
    public string userData { get; set; }
}

There can be only one detectable face (with exception, discarded for brevity) in an image.

public static async Task<string> AddPersonFace(string personGroupId, string personId, string userData)
{
    string uri = HttpHandler.BaseUri + "/" + personGroupId + "/persons/" + personId + "/persistedFaces?userData=" + userData + "&userData=" + userData;
    string jsonString = "{\"url\":\"" + HttpHandler.storagePath + "originals/" + userData + "\"}";
    HttpContent content = new StringContent(jsonString, Encoding.UTF8, "application/json");
    
    HttpResponseMessage response = await HttpHandler.client.PostAsync(uri, content);
    response.EnsureSuccessStatusCode();
    
    string responseBody = await response.Content.ReadAsStringAsync();
    return responseBody;
}

Training

Before we can run any comparison, we need to train our system. Training is about running an ML model on the person group that we defined.

public static async Task<string> TrainPersonGroups(string personGroupId)
{
    string uri = HttpHandler.BaseUri + "/" + personGroupId + "/train";
    
    HttpResponseMessage response = await HttpHandler.client.PostAsync(uri, null);
    response.EnsureSuccessStatusCode();
    
    string responseBody = await response.Content.ReadAsStringAsync();
    return responseBody;
}

It is going to generate an internal feature base and associate with different objects. The Faces we associated with each person, will have some extracted data persisted. This does not mean the whole image but a feature set that ML would use to run comparison against.

Training status can be queried, once completed we can run our comparison query!

public static async Task<string> StatusPersonGroups(string personGroupId)
{
    string uri = HttpHandler.BaseUri + "/" + personGroupId + "/training";
    
    HttpResponseMessage response = await HttpHandler.client.GetAsync(uri);
    response.EnsureSuccessStatusCode();
    
    string responseBody = await response.Content.ReadAsStringAsync();
    return responseBody;
}

Searching

Now our Machine Learning setup is populated with data and trained. Let’s query for a face to identify a person!

The steps are simple

• Upload candidate image to a blob store. I have explained the process earlier.

• Find FaceIds of all the faces in the candidate image

• For all FaceIds in the candidate image, make an identification query

• Identification query returns candidates with PersonId and confidence level

• Resolve PersonId to Person Name

A few classes to represent the data we will be sending and receiving:

 public class FaceRectangle
        {
            public int top { get; set; }
            public int left { get; set; }
            public int width { get; set; }
            public int height { get; set; }
        }
        public class Visitors
        {
            public string faceId { get; set; }
            public FaceRectangle faceRectangle { get; set; }
        }
        public class FaceQueryPayload
        {
            public string personGroupId { get; set; }
            public List faceIds { get; set; }
            public int maxNumOfCandidatesReturned { get; set; }
            public double confidenceThreshold { get; set; }
        }
        public class Candidate
        {
            public string personId { get; set; }
            public double confidence { get; set; }
        }
        public class CandidateObject
        {
            public string faceId { get; set; }
            public List candidates { get; set; }
        }

Upload candidate image to blob storage

 private async void btnTestPic_Click(object sender, RoutedEventArgs e)
        {
            btnTestPic.IsEnabled = false;
            FileOpenPicker filePicker = new FileOpenPicker();
            filePicker.SuggestedStartLocation = PickerLocationId.PicturesLibrary;
            filePicker.ViewMode = PickerViewMode.Thumbnail;
            filePicker.FileTypeFilter.Clear();
            filePicker.FileTypeFilter.Add(".jpeg"); filePicker.FileTypeFilter.Add(".jpg");
            filePicker.FileTypeFilter.Add(".png"); filePicker.FileTypeFilter.Add(".gif");
            StorageFile file = await filePicker.PickSingleFileAsync();
            
            CloudBlockBlob blob = null;
            string blobFileName = null;
            if (null != file)
            {
                BitmapImage bitmapImage = new BitmapImage();
                IRandomAccessStream fileStream = await file.OpenAsync(FileAccessMode.Read);
                bitmapImage.SetSource(fileStream);
                CapturedPhoto.Source = bitmapImage;
                CapturedPhoto.Tag = file.Path;
                blobFileName = System.Guid.NewGuid() + "." + file.Name.Split('.').Last();
                await HttpHandler.tempContainer.CreateIfNotExistsAsync();
                BlobContainerPermissions permissions = new BlobContainerPermissions();
                permissions.PublicAccess = BlobContainerPublicAccessType.Blob;
                await HttpHandler.tempContainer.SetPermissionsAsync(permissions);
                blob = HttpHandler.tempContainer.GetBlockBlobReference(blobFileName);
                await blob.DeleteIfExistsAsync();
                await blob.UploadFromFileAsync(file);
                string uri = "https://api.projectoxford.ai/face/v1.0/detect?returnFaceId=true";
                string jsonString = "{\"url\":\"" + HttpHandler.storagePath + "visitors/" + blobFileName + "\"}";
                HttpContent content = new StringContent(jsonString, Encoding.UTF8, "application/json");
                HttpResponseMessage response = await HttpHandler.client.PostAsync(uri, content);
                response.EnsureSuccessStatusCode();
                string responseBody = await response.Content.ReadAsStringAsync();
                List names = await VisitorCmds.CheckFace(responseBody, ((PersonGroups)cmbPersonGroup.SelectedItem).personGroupId);
                txtResponse.Text = string.Join(",", names.ToArray());
            }
            btnTestPic.IsEnabled = true;
        }
    }

The Face Comparison

 public static async Task<List> CheckFace(string responseString, string personGroupId)
        {
            List visitors = JsonConvert.DeserializeObject<List>(responseString);
            List faceIds = new List(); ;
            foreach (Visitors visitor in visitors)
            {
                faceIds.Add(visitor.faceId);
            }
            FaceQueryPayload jsonPayLoad = new PhotoBooth.VisitorCmds.FaceQueryPayload();
            jsonPayLoad.personGroupId = personGroupId;
            jsonPayLoad.faceIds = faceIds;
            jsonPayLoad.maxNumOfCandidatesReturned = 1;
            jsonPayLoad.confidenceThreshold = 0.5;
            string uri = "https://api.projectoxford.ai/face/v1.0/identify";
            string jsonString = JsonConvert.SerializeObject(jsonPayLoad);
            HttpContent content = new StringContent(jsonString, Encoding.UTF8, "application/json");
            HttpResponseMessage response = await HttpHandler.client.PostAsync(uri, content);
            response.EnsureSuccessStatusCode();
            string responseBody = await response.Content.ReadAsStringAsync();
            List candidates = JsonConvert.DeserializeObject<List>(responseBody);
            List names = new List();
            foreach (CandidateObject candidate in candidates)
            {
                //get person from personId
                if (candidate.candidates.Count != 0)
                {
                    string name = await PersonCmds.GetPerson(personGroupId, candidate.candidates[0].personId);
                    names.Add(name);
                }
            }
            return names;
        }
    }

I’ll publish complete code on github shortly after cleanig and some error handling. Till then, Hack It!