Omega2 AWS IoT Temperature Sensor Monitor

Intro

The guide focuses on connecting the Omega to AWS IoT, we assume that you’re already familiar with how MQTT works and how AWS IoT works, specifically Device Shadows. We recommend taking a peek at the AWS IoT overview if you’re not yet familiar: https://aws.amazon.com/iot-core/. Further, we'll connect a 1-wire temperature sensor, register the sensor at boot and write the script to publsih the sensor readings to AWS IoT. Finally, we will use AWS to visualize the incoming sensor data. We’ll first create an ElasticSearch cluster to store and index the sensor data, make an AWS IoT rule that will forward the sensor readings to our new ElasticSearch cluster, and use Kibana to actually visualize the readings.

A small note before we proceed: AWS is a paid service and your usage may incur fees. Use at your own discretion!

Setting up AWS IoT

You’ll need an Amazon AWS account, if you don’t have one already, go to https://aws.amazon.com/ and sign up.

Then navigate to the AWS IoT service, you’ll be greeted with something like:

Hit the Register a thing button and we’ll get started.

We’ll only need a single Thing for now so, hit Create a single thing. You’ll then be prompted to give your new Thing a name, we decided to name it after the Omega we’re using:

For now, you can skip the rest of the settings, you can always go back and edit them later. Scroll to the bottom and hit Next. Our next step will be to create a means of authenticating your Omega’s communication with AWS IoT. We’ll use their very nice One-Click certificate creation:

This will generate a certificate, and public and private keys. You’ll want to download the certificate and private key to your computer, then hit Activate so the certificate is enabled:

Once you hit Attach a policy, you’ll be taken to an empty policy screen. It’s a little odd that you can’t create a policy right here but that’s ok, hit Register Thing and we’ll circle back.

Now our Thing has been created and we’re back to the main AWS IoT console. We still need to create a policy to give permission to the certificate we just created, so navigate to the Secure menu and then the Policies sub-menu, and hit Create Policy:

Give your policy a name (we chose to name ours something super creative: omega-thing-policy) and hit Advanced Modeto copy our policy in. Before you do that, note that you’ll have to modify our policy to suit your AWS account and the AWS region you’re using. Replace [YOUR AWS REGION] with the region you’re using (we used us-east-2 for this guide) and replace [YOUR AWS ACCOUNT NUMBER] with … your AWS account number, it will be visible on the upper right side of the AWS console, next to the Support and Region drop-downs.

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Effect": "Allow",
     "Action": "iot:Connect",
     "Resource": "arn:aws:iot:[YOUR AWS REGION]:[YOUR AWS ACCOUNT NUMBER]:client/${iot:ClientId}"
   },
   {
     "Effect": "Allow",
     "Action": [
       "iot:Receive",
       "iot:Publish"
     ],
     "Resource": [
       "arn:aws:iot:[YOUR AWS REGION]:[YOUR AWS ACCOUNT NUMBER]:topic/$aws/things/${iot:Connection.Thing.ThingName}/shadow/*"
     ]
   },
   {
     "Effect": "Allow",
     "Action": [
       "iot:Subscribe"
     ],
     "Resource": [
       "arn:aws:iot:[YOUR AWS REGION]:[YOUR AWS ACCOUNT NUMBER]:topicfilter/$aws/things/${iot:Connection.Thing.ThingName}/shadow/update/*",
       "arn:aws:iot:[YOUR AWS REGION]:[YOUR AWS ACCOUNT NUMBER]:topicfilter/$aws/things/${iot:Connection.Thing.ThingName}/shadow/get/*"
     ]
   },
   {
     "Effect": "Allow",
     "Action": [
       "iot:GetThingShadow",
       "iot:UpdateThingShadow"
     ],
     "Resource": [
       "arn:aws:iot:[YOUR AWS REGION]:[YOUR AWS ACCOUNT NUMBER]:thing/${iot:Connection.Thing.ThingName}"
     ]
   }
 ]
}

The policy needs to be attached to our certificate. In general, the permissions of devices that authenticate with a certificate are set with the attached policy. Our policy is made to be generic and reusable, so you can have many certificates and use this policy with all of them

For the curious, here’s a brief overview of the permissions set by our policy:

• Allows MQTT connections from clients with a client ID that matches the Thing name

• A small but important note, this means only a single MQTT connection to AWS IoT can be made with this certificate

• Allows the connected device to receive messages from and publish to all MQTT topics under your Thing’s shadow

• Allows the connected device to subscribe to all of the shadow/update and shadow/get topics for the Thing

• Allows the connected device to retrieve and update the Thing’s shadow

Now that your policy is created, we’ll need to attach it to the certificate. You should have been taken back to the Policies view, click on the Certificates sub-menu and then on your certificate:

In the certificate view, click the Actions drop-down menu and then Attach Policy:

Select the omega-thing-policy and hit Attach:

Ok, we’ve set everything up on AWS IoT, let’s move to the Omega!

Setting up the Omega

First, you’ll need to make sure your Omega is connected to a WiFi with internet and is on the latest firmware. See our first time setup guide for details.

Next, we’ll need to transfer the certificate and private key files to the Omega. We used the rsync command to transfer the files to the /root directory on the Omega, but there’s a variety of ways to accomplish this, see our guide on transferring files to the Omega for more details.

Now we’re ready to download and run the script:

wget -qP /tmp https://raw.githubusercontent.com/OnionIoT/Onion-Scripts/master/awsiot_setup.sh; sh /tmp/awsiot_setup.sh ; rm /tmp/awsiot_setup.sh

This will install the required Mosquitto MQTT tools and take input from the user to configure the MQTT connection.

The first required input is the AWS IoT endpoint with which the MQTT connection will be made. This IoT endpoint is unique and specific to every AWS account. To find this for your account, go to your Thing in the IoT console, select the Interact sub-menu and copy the endpoint under Rest API Endpoint:

The rest of the inputs are fairly straightforward:

• Enter the Thing name, Omega-27BF in this case

• Then the path to the certificate file

• And finally the path to the private key file

This will configure the MQTT connection and output some examples for subscribing and publishing to AWS IoT shadow topics:

This last bit is pretty useful, so it will be copied to a file: /root/aws-topics.txt. So now you’re all set up and connected to AWS IoT!

Testing the Connection

Let’s test our connection by updating the Thing’s device shadow using the Omega:

mosquitto_pub -t \$aws/things/Omega-27BF/shadow/update -m '{"state": {"desired": {"temperature": 1 }}}' -q 1

If everything has been configured correctly, you can see that your Thing’s shadow has been updated on the AWS IoT console:

Pushing Sensor Reading to AWS IoT at an interval

The next step is to push sensor reading to AWS IoT at an interval. Our sensor of choice is a 1-Wire temperature probe, so we will cover how to wire up the sensor, configure the Omega to register the sensor at boot, and set up a script to publish the sensor readings to AWS IoT.

Hardware

A pretty quick and easy setup of the hardware, we’ll be making the following connections:

• Connect the sensor’s Ground (Black wire) to GND on the Omega

• Pull-up resistor: connect the sensor’s Vcc (red wire) and Data line (yellow wire) with the resistor

• Connect the sensor’s Data line (yellow wire) to GPIO18 on the Omega

• Connect the sensor’s Vcc (red wire) to 3.3V on the Omega

Sensor Setup

Ok, moving on to the software. All of the relevant code can be found in our aws-temperature-sensor repo on GitHub.

To start, we’ll setup our Omega to automatically configure a 1-wire bus master on GPIO18 (where we’ve plugged in our temperature sensor) at boot time. We’ll do that by adding a file to /etc/modules.d. While the Omega is booting, it will look at that directory for kernel modules that should be loaded before the boot process is completed.

So let’s make /etc/modules.d/w1-gpio-custom and populate it with the following:

w1-gpio-custom bus0=0,18,0

Now reboot your Omega, and if everything went as it should, there should be a /sys/devices/w1_bus_master1/directory.

Scripting

Next we’ll need a script that will:

• Read the sensor and format the output

• Generate an AWS IoT-compatible JSON string

• Finally, use MQTT to update the AWS IoT Thing’s shadow

We’ll run the following command to download the script:

cd /root; wget 

To run the script, you’ll need to specify your ‘Thing’ name as a command line argument. So if your Thing is named Omega-27BF, the script would be called like this:

sh publishReading.sh Omega-27BF

If you’re interested in learning how the script works, take a look at the code on GitHub.

Automating the Push

Finally, let’s setup a cron job to run this script to publish the temperature reading to AWS IoT every 5 minutes: run crontab -e and paste in the following but replace [THING NAME] with your actual Thing name:

#
*/5 * * * * sh /root/publishReading.sh [THING NAME]
#

Restart the cron daemon, /etc/init.d/cron restart and you’re Omega will send the sensor reading to AWS every 5 minutes.

Visualizing Sensor Data on AWS

We'll use AWS to visualize the incoming sensor data. To accomplish this, we’ll first create an ElasticSearch cluster to store and index the sensor data, then we’ll make an AWS IoT rule that will forward the sensor readings to our new ElasticSearch cluster, and finally, we’ll use Kibana to actually visualize the readings.

Creating an ElasticSearch Domain

Let’s log in to AWS and select the ElasticSearch service, then create a new domain:

You’ll need to give your domain a name, we’re following our streak of creative naming, and have decided to call our domain iot-dashboard:

The next step is configuration of the ElasticSearch cluster itself. For our purposes, the default settings will do just fine. Scroll to the bottom and hit Next.

Next, we’ll configure network access. Since we want to make the ElasticSearch domain accessible by the outside world (that is, any browser) and not just by internal AWS services, we’ll select Public Access:

Now, we need to configure access to the domain. For our example, we’re setting the access to be fully open:

An important note: Open access to the domain is not recommended in any production environment or with sensitive data. This was done just for illustrative purposes and we encourage you to look into more secure domain access policies.

Review your configuration and hit Confirm to create the domain. It will take a few minutes for the domain to be configured and to start running. In the meantime, you can check out your newly created ElasticSearch domain, Make a note of the endpoint, we’ll be using that in a minute.

In an ElasticSearch domain, data with similar characteristic is stored (and can then be searched) under an index. A single ElasticSearch domain can have as many indexes as you want. Data points that are meant to be stored in the index are contained in a document that is expressed in JSON. Each In our case, we’ll have an index to store temperature data from our sensor, with each sensor reading in an individual document. The final piece of the puzzle is a mapping that defines how a document and its fields are stored and indexed. The mapping is usually specified when the index is created.

We can create a new index and mapping in our ElasticSearch domain with an HTTP PUT request. You can install curlon your Omega:

opkg update
opkg install curl

Run the following code from your Omega, just make sure to replace the first part of the endpoint at the bottom to your ElasticSearch domain endpoint:

curl -i -H'Content-Type: application/json' -X PUT
  -d
'{
 "mappings": {
   "temperature": {
     "properties": {
       "timestamp": {
         "type": "long",
         "copy_to": "datetime"
       },
       "datetime": {
         "type": "date",
         "store": true
       },
       "temperature": {
         "type": "float"
       }
     }
   }
 }
}
'
'https://search-iot-dashboard-ogqgvki7phpnbtfrjtyoi6bt7e.us-east-2.es.amazonaws.com/temperature' 

This will create an index named temperature and setup a mapping for the date & time and the temperature reading.

Forwarding Data from AWS IoT to the ElasticSearch Domain

Now that the ElasticSearch domain and index are up and running, we’ll want to setup our IoT Thing to automatically populate the index with every sensor reading. To do that, we’ll make an AWS IoT rule. First navigate to the AWS IoT service, then to the Act menu, and then hit the Create a rule button:

Give your rule a descriptive name:

Scroll down and configure the message source. We’ll want all messages from accepted shadow updates of last week’s Omega-27BF Thing, so we set the topic filter to:

$aws/things/Omega-27BF/shadow/update/accepted

For maximum flexibility, the rule should forward the whole shadow update and the associate timestamp, so the attribute should be set to:

*, timestamp() AS timestamp

Notice how the Rule query statement automatically updates:

Now we’ll need to set an action for the Rule. Scroll down and hit Add action:

As the action, select Send messages to the Amazon ElasticSearch Service and hit Configure Action:

On the new page, select a Domain name from the drop-down. We’re selecting iot-dashboard, the domain we created earlier:

That will automatically populate the Endpoint form. The final three inputs define how each new document will be created: the document will be posted to the temperature index we created earlier, it will be of a type temperature, and each new document will be given a unique ID by generating a uuid:

And finally, we’ll need to give AWS IoT permission to access the ElasticSearch resource. Hit Create a new role and AWS will make a new role and configure the permissions so that everything is good to go. We named our new role iot-es-access to make it descriptive:

Scroll down and hit Add Action. This will take you back to the Rule page, scroll down and hit Create rule:

Your rule is now setup and all shadow updates of the Omega-27BF device will be added to the temperature ElasticSearch index. If you followed last week’s guide, this means every five minutes the reading will be transmitted to AWS IoT and then your ElasticSearch index!

To ensure the data is going through, wait about half an hour and then use your Omega to query the ElasticSearch index:

curl -i -X GET \  'https://search-iot-dashboard-ogqgvki7phpnbtfrjtyoi6bt7e.us-east-2.es.amazonaws.com/temperature/_search'

Again, make sure you replace the endpoint with your own ElasticSearch endpoint. The response should look something like:

{
 "took": 16,
 "timed_out": false,
 "_shards": {
   "total": 5,
   "successful": 5,
   "skipped": 0,
   "failed": 0
 },
 "hits": {
   "total": 5,
   ...
 }
}

Pay special attention to hits.total since it represents the number of datapoints stored in the ElasticSearch index.

Visualizing Data

And now, the fruit of our labors: visualizing the data we’ve been collecting. We’ll use another tool from the Elastic stack called Kibana to help us visualize the data. Kibana can be used to make really wicked dashboards, so we definitely recommend learning more about it if you’re a fan of visualizing data.

Head back to the ElasticSearch Service and click on your domain, then click on the URL labelled Kibana:

This will open the Kibana, a tool meant for visualizing and exploring ElasticSearch data, in a new browser tab:

Go to the Visualize menu, and then hit the + button to Create a New Visualization. We’ll select a Line chart:

Select the temperature index:

We’ll want to setup the Y-Axis to show an average of the temperature reported by the Omega. Hit the > button beside Y-Axis, then change the Aggregation to Average, and select the field to be state.reported.temperature (from our device shadow updates):

Scroll down and we’ll configure the X-Axis to show the date and time in a linear fashion. Select X-Axis as the buckets type, then Date Histogram for the Aggregation. The Field should be set to datetime and the Interval to auto:

Finally, hit the large > button for the changes to take effect:

Finally, your visualization is now up:

Hit the save button, give the visualization a name, and Save it:

Now your temperature visualization will be always accessible from the Visualize menu. As you collect more and more data, you might want to change the Time Range of the data in your visualization:

And there you have it, an awesome, online visualization of sensor data coming from your Omega!