User Activity Recognition with Azure

General Idea

User Activity Recognition with Azure is a mobile web application that gathers data from the accelerometer, sends them to Azure IoT Hub and tells if the user is standing still or moving. Finally, you can visualized all the data in a Node.js application that runs in local.

The project consists of two separate parts: cloud-based and edge-based implementation. In cloud-based implementation the data of the accelerometer are sent to the hub and the computation of the user activity is at cloud level. In edge-based implementation, instead, the computation is done in the webapp and only the state of the activity is sent to the hub.

The implementation part of this project takes for granted that you already have an Azure IoT Hub, at least a device and the connection strings. You can check this tutorial to see how to configure them.

Components

Azure IoT Hub

The IoT hub will be used as a MQTT broker, but it's not complete and it doesn't support all the standard behaviours.

Azure Web App

Azure web app hosts the web applications, so that we can run them from any internet browser.

LocalNode.jsapp for data visualization

Node.js is used to run an application that lets you visualize the data in local. You have to install also the package @azure/event-hubs.

Cloud-based implementation

In cloud-based implementation the user activity recognition is at cloud level.

• Azure Web App

The web app is composed of three modules: server.js launches the site and sends the messages with the data to the hub using MQTT protocol; index.html is the what we actually see when we run the we app; accelerometer.js collects data from the accelerometer of the smartphone and sends them to server.js using a websocket; web.config is the configuration file of the webapp,

The server file must contain some requirements to communicate with the hub using MQTT protocol and with accelerometer.js using socket. Then we can launch the public folder that contains index.html and accelerometer.js. Index.html is the main file that will load the other one. A connection via socket is established in the server, so that when a message is sent from the webapp, it is forwarded to the hub. It requires also connection string of a device.

const express = require('express');
const http = require('http');
const WebSocket = require('ws');
const path = require('path');
const EventHubReader = require('./scripts/event-hub-reader.js');

The following function of the sever gets the messages from accelerometer.js and sends them to the hub in a JSON string.

socket.on('accelerometer', function(message){
  var JSONMessage = new Message(JSON.stringify({
    state: message.state
  }));
  //sending message to azure iot hub
  client.sendEvent(JSONMessage, function (err) {
    if (err) {
      console.error('Error while sending the message to the hub: ' + err.toString());
    } else {
      console.log('Messages sent');
    }
  });
});

The webapp is very simple, so the html file contains only a title and 3 spans to write the values of the accelerometer on X, Y and Z axes. It must contain scripts for calling accelerometer.js, a script to use the sockets and a script to use jquery.

<script src="/socket.io/socket.io.js"></script>
<script src="https://code.jquery.com/jquery-3.4.0.min.js" integrity="sha256-BJeo0qm959uMBGb65z40ejJYGSgR7REI4+CW1fNKwOg=" crossorigin="anonymous" type="text/javascript" charset="utf-8"></script>
<script src="accelerometer.js" type="text/javascript" charset="utf-8"></script>

The main file is accelerometer.js and uses the accelerometer of generic sensor API with a frequency of 1 Hz. It starts the accelerometer of the smartphone and sends the latest values to the hub and to the webapp, so that we can visualize them. Messages are not sent every 1 second because Azure has a limit of 8000 messages per day.

setInterval(function(){
    var message = {
    xaxis: x,
    yaxis: y,
    zaxis: z
    };
    socket.emit("accelerometer",message);
}, 5000);

Web.config is the configuration file of the webapp that specifies that the module to launch is server.js.

Now that all the files are ready, we can create a web app on Azure website. We can specify the parameters like the subscription, the name of the webapp, the operating system that we are using and the typology of the code (e.g..NET, nodejs, etc). Finally, press create button to create the webapp.

Remember to enable the Web Socket in Configurations and enable the HTTPS protocol in TLS/SSL settings. The webapp will not work without them.

Now go to the folder that contains all our files and write in the terminal

npm start

Then open it in VisualStudioCode simply writing in the terminal:

code .

We need to install the Azure webapp extension of VSCode and sign in to the microsoft account.

In the Azure extension part of VSCode, select the blue arrow and load the folder. Then, select the webapp where you want to upload the files and finally select deploy. Now you will see all your files in the Files folder of the webapp.

You can go to the site https://name-of-your-webapp.azurewebsites.net to visualize your app. In my case https://clouduseractivityrecognition.azurewebsites.net/.Of course it works only on the smartphones, because PCs don't have the accelerometer.

Local Node.js app for data visualization

The nodejs app runs in local and let us visualize the data that arrive at the hub. If the difference between the previous and the latest value received for any axis is not comprised between 0.6 and -0.6 the user is moving, otherwise he is standing still. The latest values arrived are visualized in spans, while the values of the last hour are visualized in charts.

This part of code needs the connection string of the hub and the consumer group, in my code they are set as environment parameters.

Go to the folder and run the webapp typing in the terminal:

npm start

You can run in your browser http://localhost:3000 to visualize it.

As alternative, you can create another web app with this code and run it as explained in the previous section. In this case you should also handle the sicurity issues, so that not everyone can access your private data.

In the previous images we can see in the terminal the messages that arrive at the hub and that are later visualized in a local site. On the left there is the name of the device that is sending data, then in the central part there are 3 labels, one for each axis. In the charts we can visualize the data and the activities of the last hour.

Edge-based implementation

In edge-based implementation the user activity recognition is at edge-level.

• Azure Web App

The modules of the web app are the same as before, but now their the logic changes.

Server.js sends to the hub the latest activity and not the values of the accelerometer.

Index.html contains also a span to display the latest activity of the user.

Accelerometer.js start the accelerometer and, if the difference between the latest and the previous value is comprised between -0.6 and 0.6, it display that the user is standing still, otherwise the user is moving. Finally data are sent to the server via socket, forwarded to the hub and displayed in the local nodejs webapp,

You can access the site running in the browser https://edgeuseractivityrecognition.azurewebsites.net/. Of course it works only on the smartphones, because PCs don't have the accelerometer.

Local Node.js app for data visualization

The Node.js app is similar to the one of the cloud-based implementation, but this time it only visualizes the data that are sent to the hub, without any computation.

You can start it with the command:

npm start

and then run http://localhost:3000 in the web browser.

In the previous images we can visualize the messages that arrive at the hub later visualized in a local web app. In this type of implementation we can only visualize the activities of the user and not the data of the sensor.

Resources

Generic sensor API

Create a Node.js web app in Azure

Azure IoT samples for Node.js

Real-time data visualization in web apps

Example of web apps