Arduino/Genuino 101: Build an Activity Recognition Device

Our Team

We are Andrea Napoletani, Alessandro Giannetti and Riccardo Pattuglia, students of the Master Course of Engineering in Computer Science at "La Sapienza" University of Rome.

The goal of this project is to build up a simple Activity Recognition device able to recognize four kinds of everyday activities: walking, sitting, standing still and running.

An overview of your daily activities is available at your personal page on Redash.

Useful Links

• LinkedIn profiles: Andrea Napoletani, Alessandro Giannetti, Riccardo Pattuglia
• GitHub project page
• Proof of Concept presentation

Project Overview

The project is divided into five main steps:

• Train the Genuino 101 board on some regular patterns (referred to each activity) in order to allow the classification of future data captured by the accelerometer.
• Send data from Genuino 101 to the Android App through Bluetooth Low Energy (BLE) technology.
• Connect the Android App to AWS IoT service and send data exploiting MQTT protocol.
• Connect and store data from AWS IoT to AWS DynamoDB.
• Query the AWS DynamoDB from Redash to plot the activity overview.

Step 1: Train Genuino 101

The Genuino 101 has been chosen for this project due to the integrated Intel Curie Module which features a Pattern Matching Engine for the classification of the data given in input and a Bluetooth Low Energy to establish a connection with an Android smartphone.

For the implementation of the project, we have used the libraries provided by the producer.

In order to let Genuino 101 be able to classify new instances of data collected by the accelerometer, it has to be trained with significant data. For this reason, we have collected the data from the device during the execution of the different activities that we want to recognize and then we have integrated this data into the sketch in order to obtain a pre-trained model.

It is very important to remember that the Genuino 101 must be placed into the left pocket in order to be able to successfully reuse the data given inside the attached file, otherwise, you should retrieve new data to retrain the model. The data can be retrieved directly from the Serial Monitor and then replaced into the sketch (the ability of printing data on the serial monitor is already included in the program).

The data stored into the memory of the device must not be loaded into RAM otherwise the device will behave abnormally and probably will stop working, for this reason, we used the PROGMEM library well documented here, making possible to train the Pattern Matching Engine without wasting the limited RAM of the board.

This is an example of 128 bytes of data for one neuron of the Intel Curie Module:

static byte subject102_1_0[] PROGMEM = {60, 60, 60, 60, 60, 60, 59, 59, 59, 59, 59, 60, 60, 59, 60, 59, 59, 60, 59, 60, 60, 59, 60, 60, 59, 60, 59, 60, 60, 59, 60, 60, 59, 60, 60, 60, 59, 59, 59, 59, 59, 60, 60, 59, 59, 59, 60, 60, 60, 59, 59, 59, 60, 60, 59, 59, 59, 59, 59, 60, 60, 59, 60, 60, 60, 59, 59, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 59, 60, 60, 60, 60, 60, 60, 60, 60, 59, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 59, 60, 60, 60, 60, 59, 59, 59, 59, 58, 60, 60, 59, 59, 59, 59, 59, 59, 59, 60};
CuriePME.learn(subject102_1_0, 120, 1); 

As you can see, the original raw values produced by the accelerometer are rescaled in a range between 0 and 255, to allow the storage of more values into the array (one byte against 4 bytes of the original value). And the only axis considered is the X AXIS as we are focusing only on the vertical movements (given the fact that the board is inside the pocket the vertical axis is not Z).

CurieIMU.readAccelerometer(raw[0], raw[1], raw[2]);
     /* Map raw values to 0-255 */
     accel[i] = (byte) map(raw[0], IMULow, IMUHigh, 0, 255); 

Due to this fact, Genuino 101 might return an "unknown" detection if the board is not perfectly vertical inside the pocket or it moves too much inside it.

After the training phase, we can go through the classification one. This part is simply done giving the data captured at runtime to the classification function of the Genuino 101 using the RBF classifier (kNN was inaccurate):

readVectorFromIMU(vector);
     category = CuriePME.classify(vector, 120);
     switch (category) {
       case 1: ... ; break;
       case 2: ... ; break;
       case 3: ... ; break;
       case 4: ... ; break;
       default: ... ; break;
     } 

Despite the limited amount of memory for the training, the results obtained were pretty good.

Step 2: Connect Genuino 101 to the Android App

After the classification, the second step is to send the result to the Android device.

For this purpose we have used the BLE libraries provided by Intel specifically for Genuino 101.

We first set up the Bluetooth connection parameters, using a personal UUID:

BLEPeripheral blePeripheral;
BLEService fitnessService("19B10000-E8F2-537E-4F6C-D104768A1214");
BLEUnsignedCharCharacteristic fitnessTypeChar("19B10001-E8F2-537E-4F6C-D104768A1214", BLERead | BLENotify);

Then in the setup function we set the BLE up and start to listen for connections:

 blePeripheral.setLocalName("Activity");
 blePeripheral.setAdvertisedServiceUuid(fitnessService.uuid());
  
 blePeripheral.addAttribute(fitnessService);
 blePeripheral.addAttribute(fitnessTypeChar);
  
 blePeripheral.begin();

After that the work proceeds on the Android device.

For this project we have used the Bluetoothlegatt android app by google.

Starting from this base, we have integrated the AWS services and the interpretation of the data sent by Genuino 101.

Genuino 101 sends only one integer which defines the category associated to the activity recognized so the application receives the data and transforms it in something more usefull.

      int number = 0;
               for (byte byteChar : data)
                   number = byteChar;
              
               measuraments.add(number);
               String activity = "Unknown";
               if (measuraments.size() == 5) {
                   int stills = Collections.frequency(measuraments, 1);
                   int sits = Collections.frequency(measuraments, 2);
                   int walks = Collections.frequency(measuraments, 3);
                   int runs = Collections.frequency(measuraments, 4);
                   int max = Math.max(sits, Math.max(Math.max(stills, walks), runs));
                   if (max == stills)
                       activity = "Staying still";
                   if (max == runs)
                       activity = "Running";
                   if (max == walks)
                       activity = "Walking";
                   if (max == sits)
                       activity = "Sitting";
                   if (max == 0)
                       activity = "Unknown";
                   measuraments.clear();
                   AWSServiceClient.getInstance().publish(activity);
               }
               switch (number) {//Integer.parse(stringBuilder.toString())) {
                   case 1:
                       activity = "Staying still";
                       break;
                   case 2:
                       activity = "Sitting";
                       break;
                   case 3:
                       activity = "Walking";
                       break;
                   case 4:
                       activity = "Running";
                       break;
                   default:
                       activity = "Unknown";
                       break;
               }
               intent.putExtra(EXTRA_DATA, new String(data) + "\n" + activity);
           }

As you can see there is also an implementation of a list, which keeps track of the last five values and returns the most frequent one. This is done to avoid false values and to avoid the "unknown" states that are produced due to the low precision of the Pattern Matching Engine of Genuino 101.

Step 3: AWS IoT Configuration and Connection with Android

This part has been developed exploiting this example available on awslabs GitHub profile.

Here we will connect our Android Application to Amazon Web Service IoT platform in order to send data from the application (coming from Genuino 101) to our IoT Console using MQTT protocol.

We will create a subscription my/iotminiproject needed to send JSON format messages of this form:

{    
    "number": "Identifier of the activity",
    "type": "Activity Type"
}

To connect your Android App to AWS IoT you will need Amazon Cognito to authorize a WebSocket connection.

AWS Console Part

• Here you can create an account and access to your Amazon Cognito console. Go to Manage Identity Pools and Create new identity pool.
• Give your identity pool a name and ensure that Enable access to unauthenticated identities section is checked. This allows the application to assume the unauthenticated role associated with this identity pool. This procedure will create two roles in Identity and Access Management (IAM) called Cognito_yourRoleNameAuth_Role and Cognito_yourRoleNameUnauth_Role.
• Save the Identity pool ID value, that should look similar to: "us-east-1:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" and the region, these will be used later in the application.
• Now we will attach a policy to the unauthenticated role which has permissions to access the required AWS IoT APIs. This is done by attaching an IAM Policy to the unauthenticated role in the IAM Console Press Attach Policy button in the "Permission" tab and select AWSIoTFullAccess, this policy allows the application to perform all operation on the Amazon IoT service.
• We have to create a policy on AWS IoT Console to allow connecting to AWS IoT as well as allowing publishing, subscribing and receiving messages on whatever topics you will use in the same application. Click on Secure - Policies - Advanced Mode and add the following policy (you can find your own ARN on the top of this page):

{  
    "Version": "2012-10-17",
    "Statement": [
    {      
        "Effect": "Allow",
        "Action": "iot:Connect",
        "Resource": "YOUR_POLICY_ARN"    
    },
    {
        "Effect": "Allow",
        "Action": [
            "iot:Publish",
            "iot:Subscribe",
            "iot:Receive"
          ],
        "Resource": "YOUR_POLICY_ARN"    
    }
  ]
}

Android App Part

• Open awsconfiguration.json file in res - raw and update the following constants with the appropriate values:

"CredentialsProvider": {
    "CognitoIdentity": {
        "Default": {
        "PoolId": "YOUR_POOL_ID",
        "Region": "YOUR_REGION"
        }
    }
}

• Open PubSubActivity.java and update the following constants with the appropriate values:

CUSTOMER_SPECIFIC_ENDPOINT = "YOUR_ENDPOINT";

You can find your Endpoint in your AWS IoT Console at Settings panel.

Your AWS IoT is configured, here we can see the number of connections with our devices and the MQTT exchanged messages.

Step 4: From AWS IoT to AWS DynamoDB

This part has been developed exploiting this guide available on AWS Documentation.

Now that the connection between Genuino 101 and AWS IoT is made we want a way to store our messages; for this purpose, we are going to use AWS DynamoDB.

We have to create a rule that allows us to take information from an incoming MQTT message and write it to a DynamoDB table.

• From the AWS IoT Console go to Act - Create to create a new rule. Decide a Name and a Description for our rule and select the latest available version for SQL. For Rule query statement enter:

SELECT * FROM 'my/iotminiproject'

So we have:

• Open Select an action page, choose Insert a message into a DynamoDB table and then choose Configure action.

Now we have to create a new table in our DynamoDB pressing on Create a new resource:

• Insert a name for our table and configure it as the following image:

• In Table name select the table created above. Fill the two fields Partition key value and Sort key value as the following image:

• Select Create rule to confirm the operation and start inserting data into the table.

Step 5: Analyze Your Activities on Redash

The first thing you'll want to do is connect a data source (see supported data sources). You can open the Data Sources management page by clicking the Settings icon:

Add DynamoDB Data Source

• Open Settings page, choose Data Sources and then choose DynamoDB(with DQL).

• Enter the Name of the DynamoDB table in the Name field
• Access key and private key can be found on the Amazon AWS AIM panel.
• Enter in the AIM panel and Select a user.
• Select add permissions and add all the permission for DynamoDB

• Done this you can recover the two keys in Security Credentials by selecting Create Access Key

Write a Query

• Once you've connected a data source, it's time to write a query: click on Create in the navigation bar, and then choose Query.See the “Writing Queries” page for detailed instructions on how to write queries.

Problems with DynamoDB:

• The QL of DynamoDB don’t support the aggregation functions like sum,avg, complex count and specific operation for the dates like format.

Solution to the problem:

• Redash allows you to query results from other queries

Querying Existing Query Results

The Query Results Data Source lets you run queries against results from your other Data Sources. Use it to join data from multiple databases or perform other kinds post-processing. Redash uses an in-memory SQLite database to make this possible. As a result, queries against large result sets may fail if Redash runs out of memory.

• You can enable Query Results under the Data Source tab of the settings menu. Setup is easy: just provide a name for the source. This is the name that will appear in the source dropdown of the query editor.
• SQLite query syntax should be familiar if you have worked with other SQLs. Here's an example query

SELECT a.name, b.count
FROM query_123 a JOIN query_456 b 
ON a.id = b.id

• Each of your existing queries constitutes its own "table" to SQLite. The table name is the string query_ concatenated with your desired Query ID, which can be found in that query's URL.
• So for example, a query with the URL https://app.redash.io/acme/queries/49588/source will have the table name query_49588 in SQLite.

Query Parameters

Unless specific to a one-time project, most queries can be reused by changing a WHERE clause (or filter block in NoSQL) to suit the present need. Yet it can be a hassle to Edit Source every time you make a minor change. Query Parameters let you insert values at run time without editing your base query. The syntax is straightforward: Redash recognizes any string between double curly braces {{ }} as a Query Parameter. Here more details

OurQueries

• Find all activities on DynamoDB

SCAN activity_type, activity_number
FROM activity_recognition

• Find all activities from DynamoDB queries selecting only the hours of one Day

SELECT activity_type,
      count(activity_type),
      activity_type
FROM query_188916
WHERE substr(activity_number,0,11) == "{{ DAY }}"
GROUP BY activity_type

• Find all the days in DynamoDB

SELECT substr(activity_number,0,11)
FROM query_188916
GROUP BY substr(activity_number,0,11)

• Find the numbers of activities grouped by the activity of one Day

SELECT activity_type,
      count(activity_type)
FROM query_188916
WHERE substr(activity_number,0,11) == "{{ DAY }}"
GROUP BY activity_type

Add Visualizations

By default, your query results (data) will appear in a simple table. Visualizations are much better to help you digest complex information, so let's visualize your data. Redash supports multiple types of visualizations so you should find one that suits your needs.

• Click the New Visualization button just above the results to select the perfect visualization for your needs. You can view more detailed instructions here.

Create a Dashboard

You can combine visualizations and text into thematic & powerful dashboards.

• Add a new dashboard by clicking on Create in the navigation bar, and then choose Dashboard.
• Dashboards are visible for your team members or they can be shared publicly. For more details, click here.

Our Analysis Dashboard

In our view, we decided to let the user select only the days present in our DynamoDB, So we inserted a parameter that allow the user to select only these days.

At the top of our dashboard, we have inserted a scatterplot, so that we can analyze the distribution of our activities throughout the selected day.

In the lower part instead, we have inserted two graphs that allow us to have an overview of the activities carried out during the day.

The pie chart allows us to have an overview of the activities carried out during the day in percentage, while the bar chart gives us more detailed information about the number of samples received by genuine throughout the day.