Build an IoT infrastructure with RIOT-OS, MQTT and AWS

Introduction

In this tutorial, I will explain how to create an IoT Infrastructure, focusing on the manage of the data from the Thing to the Website, using AWS.

This tutorial is linked with FOFIDES, a Forest Fire Detection System.

See the Video Presentation of FOFIDES on YouTube.

MQTT BRIDGE - FROM THE THING TO AWS IOT-CORE

Mosquitto Broker on your computer

This has to be executed in the device that you want to use as bridge for AWS.

You have to download mosquitto. To have more informations about this MQTT Broker go to he Github page: Here. I am working on Ubuntu, so:

sudo apt-get update

sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa

sudo apt install mosquitto mosquitto-clients

mosquitto -version

Sometimes this installation is not enough. So if you have problem, try this:

sudo -i                     
cd /etc/mosquitto
nano mosquitto.conf

Add this configuration and save it.

allow_anonymous true
listener 1883 "YOUR IP ADDRESS"
persistence_file mosquitto.db
log_dest syslog
log_dest stdout
log_dest topic
log_type error
log_type warning
log_type notice
log_type information
connection_messages true
log_timestamp true

To allow the sending of the data, you have to use a board connected to Internet and working with MQTT Protocol.

Connect Mosquitto Broker with AWS

You have to connect your computer and AWS. You can follow this guide if you need also the connection with EC2 (I have not used it): Here.

For my project, you can go directly to the section How to Configure the Bridge to AWS IoT Core. So you have to enter the following commands in your terminal:

aws configure

When you do this command, you will obtain this:

AWS Access Key ID [********************]:  
AWS Secret Access Key [********************]:  
Default region name []:  
Default output format []:

Fill the fields with your informations, looking at your AWS account. Then:

aws iot create-policy --policy-name bridgeMQTT --policy-document '{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Action": "iot:*","Resource": "*"}]}'

cd /etc/mosquitto/certs/

sudo wget https://www.amazontrust.com/repository/AmazonRootCA1.pem -O rootCA.pem

sudo aws iot create-keys-and-certificate --set-as-active --certificate-pem-outfile cert.crt --private-key-outfile private.key --public-key-outfile public.key --region us-east-1

aws iot attach-principal-policy --policy-name bridgeMQTT --principal <certificate ARN>

sudo chmod 644 private.key 

sudo chmod 644 cert.crt

aws iot describe-endpoint --endpoint-type iot:Data-ATS

sudo nano /etc/mosquitto/conf.d/bridge.conf

After this, launch the following command to restart the Mosquitto broker:

sudo service mosquitto restart

Now you have to connect the Device in AWS.

Connect the device in AWS

Go in AWS IoT-Core, click on Connect on Device and follow the instructions given by AWS; when you are on the page of Platform and SDK, choose you Operating System and select Python for AWS IoT Device SDK.

When you have completed the tutorial above, you have to change the Policy to allow the correct functioning of the Mosquitto Broker:

• Go in All Devices, Thing and select your Thing.
• At the bottom of the page, click on Certificate. Click on the long string which is your certificate.
• Now you will see the title Policies, click on your policy (this policy has been created automatically thankfull to the process we have done before).
• Now you see your policy in the Builder Format. Click on JSON. You will se this:

{
  "Version": "4013-70-97",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iot:Publish",
        "iot:Receive"
      ],
      "Resource": [
        "arn:aws:iot:us-a-1:111111111111:topic/sdk/test/java",
        "arn:aws:iot:us-a-1:111111111111:topic/sdk/test/python",
        "arn:aws:iot:us-a-1:111111111111:topic/sdk/test/js"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "iot:Subscribe",
      "Resource": [
        "arn:aws:iot:us-a-1:111111111111:topicfilter/sdk/test/java",
        "arn:aws:iot:us-a-1:111111111111:topicfilter/sdk/test/python",
        "arn:aws:iot:us-a-1:111111111111:topicfilter/sdk/test/js"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "iot:Connect",
      "Resource": [
        "arn:aws:iot:us-a-1:111111111111:client/sdk-java",
        "arn:aws:iot:us-a-1:111111111111:client/basicPubSub",
        "arn:aws:iot:us-a-1:111111111111:client/sdk-nodejs-*"
      ]
    }
  ]
}

You have to change the Policy adding two lines, in which you have to add your MQTT Topic. Remember to write the topic to use at least a '/', like MQTT/topic.

{
  "Version": "4013-70-97",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iot:Publish",
        "iot:Receive"
      ],
      "Resource": [
        "arn:aws:iot:us-a-1:111111111111:topic/sdk/test/java",
        "arn:aws:iot:us-a-1:111111111111:topic/sdk/test/python",
        "arn:aws:iot:us-a-1:111111111111:topic/sdk/test/js",
        "arn:aws:iot:us-a-1:111111111111:topic/MQTT/topic"        <---HERE
      ]
    },
    {
      "Effect": "Allow",
      "Action": "iot:Subscribe",
      "Resource": [
        "arn:aws:iot:us-a-1:111111111111:topicfilter/sdk/test/java",
        "arn:aws:iot:us-a-1:111111111111:topicfilter/sdk/test/python",
        "arn:aws:iot:us-a-1:111111111111:topicfilter/sdk/test/js",
        "arn:aws:iot:us-a-1:111111111111:topicfilter/MQTT/topic"   <---HERE
      ]
    },
    {
      "Effect": "Allow",
      "Action": "iot:Connect",
      "Resource": [
        "arn:aws:iot:us-a-1:111111111111:client/sdk-java",
        "arn:aws:iot:us-a-1:111111111111:client/basicPubSub",
        "arn:aws:iot:us-a-1:111111111111:client/sdk-nodejs-*"
      ]
    }
  ]
}

Save it, and now the policy is correctly set.

Sending messages from the Device to AWS

Now you have to use the MQTT code to send messages to AWS. Remember to put the code in a directory with the AWS IoT Device SDK downloaded before.

from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTClient
import paho.mqtt.client as mqtt
import json
import signal

def on_message(_client, _userdata, message):

    payload = json.loads(message.payload)
    
    topic = MQTT_PUB_TOPIC

    success = myMQTTClient.publish(topic, json_payload, 0)

    time.sleep(5)
    if(success):
        print("published",json_payload)

    
# On connect subscribe to topic
def on_connect(_client, _userdata, _flags, result):
    """Subscribe to input topic"""

    myMQTTClient.publish(MQTT_PUB_TOPIC, "Connection", 0)
    print("Connection Done")

    MQTT_CLIENT.subscribe(MQTT_SUB_TOPIC)
    print('Subscribed to ' + MQTT_SUB_TOPIC)

# Disconnect function
def disconnect_clients(signum, frame):
    MQTT_CLIENT.loop_stop()
    MQTT_CLIENT.disconnect()
    myMQTTClient.disconnect()
    print("Disconntection")
    exit(0)

signal.signal(signal.SIGINT, disconnect_clients)

MQTT_BROKER_ADDR = "YOUR IP ADDRES"
MQTT_BROKER_PORT = 1883
MQTT_BROKER_CLIENT_ID = "broker"
AWS_IOT_ENDPOINT ="YOUR AWS_IOT_ENDPOINT"
AWS_IOT_PORT = 8883
AWS_IOT_CLIENT_ID = "basicPubSub"
AWS_IOT_ROOT_CA = "YOUR AWS_IOT_ROOT_CA"
AWS_IOT_PRIVATE_KEY = "YOUR AWS_IOT_PRIVATE_KEY"
AWS_IOT_CERTIFICATE = "YOYR AWS_IOT_CERTIFICATE"

# For certificate based connection
myMQTTClient = AWSIoTMQTTClient(AWS_IOT_CLIENT_ID)
myMQTTClient.configureEndpoint(AWS_IOT_ENDPOINT, 8883)
myMQTTClient.configureCredentials(AWS_IOT_ROOT_CA, AWS_IOT_PRIVATE_KEY, AWS_IOT_CERTIFICATE)
myMQTTClient.configureOfflinePublishQueueing(-1)
myMQTTClient.configureDrainingFrequency(2)
myMQTTClient.configureConnectDisconnectTimeout(10)
myMQTTClient.configureMQTTOperationTimeout(5)
MQTT_SUB_TOPIC = "YOUR MQTT_SUB_TOPIC"
MQTT_PUB_TOPIC = "YOUR MQTT_PUB_TOPIC"
MQTT_CLIENT = mqtt.Client(client_id=MQTT_BROKER_CLIENT_ID)

# MQTT callback function
def main():
    MQTT_CLIENT.on_connect = on_connect
    MQTT_CLIENT.on_message = on_message
    MQTT_CLIENT.connect(MQTT_BROKER_ADDR, MQTT_BROKER_PORT)
    myMQTTClient.connect()
    MQTT_CLIENT.loop_forever() #

if __name__ == '__main__':
    main()

This script take the values sent by the ESP32s and redirect them to AWS.

Launch the script with (put the name of your file.py):

python3 file.py

Now we want to see the exchange of the messages, open another terminal:

mosquitto_pub -h YOUR IP -t 'YOUR TOPIC' -m 'Hi AWS'

To see if it is working:

• go in AWS in your Thing.
• Click on Activity and then on MQTT test client.
• Subscribe to the proper topic (MQTT_PUB_TOPIC in the case of the previous script).
• You will see the messages arriving in MQTT.

Create AWS Roles

You have to create two different Roles:

• IAM-ROLE-IOT: this is needed to use IoT-Core properly and to create the IoT-Rules. It has to have the following policies: AmazonDynamoDBFullAccess, AWSIoTRuleActions, AWSIoTDataAccess and AWSIoTConfigAccess.
• IAM-ROLE-LAMBDA: this is needed to use AWS-Amplify, API-Gateway, DynamoDB and Lambda together. It has to have the following policies: AmazonDynamoDBFullAccess, IAMFullAccess, AWSLambdaBasicExecutionRole, AmazonAPIGatewayAdministrator, AmazonAPIGatewayInvokeFullAccess, AWSLambda_FullAccess and AdministratorAccess-Amplify.

If you find difficulties, follow this AWS Tutorial.

Create DynamoDB to store data

• Create a DynamoDB (FireSensorTable in the case of FOFIDES):
• When you create it, put Timestamp as Partition Key
• select Number as type (not String)
• This table stores all the data that will arrive from the Thing (IoT Core)

IoT-Rule and Lambda to insert data in DynamoDB

Lambda

Create the lambda function to put items in the DynamoDB:

• Go to the AWS Management Console and navigate to the AWS Lambda service.
• Click on "Create function" and choose "Author from scratch."
• Provide a name for your function and select Python 3.9.
• Under "Permissions, " choose the IAM-ROLE-LAMBDA created before.
• Click on "Create function" to create your Lambda function.
• Write the code for your Lambda function to process and insert the data into DynamoDB.

import json
import boto3
import calendar
from datetime import datetime
import time

# Create a DynamoDB object using the AWS SDK
dynamodb = boto3.resource('dynamodb')

# Use the DynamoDB object to select our tables
table_fire = dynamodb.Table('FireSensorTable')

# Define the handler function that the Lambda service will use
def lambda_handler(event, context):
 # Generate a timestamp for the event
    time_now = datetime.now().strftime('%d/%m/%Y %H:%M:%S')

    current_GMT = time.gmtime()
    timestamp = calendar.timegm(current_GMT)

    # Write payload and time to the DynamoDB table using the object we instantiated and save response in a variable
    response = table_fire.put_item(
    Item={
        'Timestamp': timestamp,
        'Datetime': event['Datetime'],
        'Flame': event['Flame'],
        'CO': event['CO'],
        'Temp': event['Temp']

    })

    return {
       'statusCode': 200
    }

Change the the script according to your needs. This function in my project is FireSensorTakeFunction.

IoT-Rule

Create a IoT-Rule to insert the data, that are arriving from the Thing, in the DynamoDB. To create it:

• Go to the AWS IoT service in the AWS Management Console.
• Click on "Message Routing" in the left navigation pane and then select "Rules."
• Click on "Create" to create a new rule.
• Provide a name for your rule and add a description if desired.
• Under "Rule query statement, " enter the SQL query that matches the data you want to send to your Lambda function. For example in my case: SELECT * FROM 'MQTT/Topic'.
• Under "Set one or more actions, " click on "Add action" and select "Send a message to a Lambda function."
• Select your Lambda function from the drop-down menu.
• Click on "Create rule" to create your IoT rule.

Lambda, Api Gateway and Amplify to create the Website

Lambda

Create another Lambda Function to take the data from the dynamoDB and use it for the website.Do the same process of the step before.

import json

# import the AWS SDK
import boto3

from decimal import Decimal

# Funtion to sort by numeric timestamp
def sort_by_key(list):
    return list['Timestamp']

# Define the handler function that the Lambda service will use
def lambda_handler(event, context):

    # Create a DynamoDB object using the AWS SDK
    dynamodb = boto3.resource('dynamodb')

    # Use the DynamoDB object to select our tables
    table_fire = dynamodb.Table('FireSensorTable')

    # Retrieve tuples of our tables to return
    response_fire = table_fire.scan()

    output_fire = response_fire['Items']

    # Convert numeric values to strings
    for fire_json in output_fire:
        fire_json['Timestamp'] = str(fire_json['Timestamp'])

    # Sort by timestamp
    output_fire = sorted(output_fire, key=sort_by_key)


    return {
       'statusCode': 200,
       'body_fire': json.dumps(output_fire)
    }

The code is in the directory Code_Lambda_AWS and it is FireSensorMonitorFunction.

Api Gateway

Create the API, using API Gateway:

• Go to the AWS Management Console and navigate to the API Gateway service.
• Click on "Create API" and choose "REST API."
• Select "New API" and provide a name for your API.
• Under "Resources, " click on "Actions" and select "Create Method."
• Choose "GET" as the method type and click on the checkmark button.
• Configure the integration of the GET method: Choose "Lambda Function" as the integration type, Select the appropriate region, Choose the Lambda function you created on the step before (FireSensorMonitorFunction), Click on the checkmark button to save the integration.
• Click on the "Method Request" card and scroll down to the "HTTP Request Headers" section.
• Click on the "Add header" button and click then enable CORS.
• Click on the checkmark button to save the header.
• Click on the "Actions" dropdown button and select "Deploy API."
• Choose "New Stage" and provide a stage name. For example, dev.
• Click on deploy the API.

Now you have created an API. Create the website and connect API Gateway and Amplify.

Amplify

Go to the AWS Amplify console for your project:

• Create a new backend environment, click on New App and then on Host Web App.
• Click on Deploy without git provider.
• Create the index.html, index.js and index.css.
• Compress all the files in a zip.
• Upload it on the AWS Interface.
• Click on the Amplify Link and you will be redirected on the website.

Now you have your website. To see all the code, go here (so you do not have to build from scratch).

Thanks for Attention. I hope I was clear; if not, comment below and I help you.