Hardware components | ||||||
 |
| × | 1 | |||
Software apps and online services | ||||||
 |
| |||||
 |
| |||||
 |
| |||||
 |
| |||||
 |
| |||||
 |
| |||||
| ||||||
| ||||||
This project is centered around the fourth goal of the United Nations' Sustainable Development Goals, "Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all".
You're from the future; currently facing the next big pandemic. You're looking for a solution, and let me tell ya; you came to the right place. Here are few things we tried back in 2020 (not in any particular order):
Burning down 5G towers
Buying all the toilet paper
Dispensing wisdom
Using AntiVirus
Switching to Macintosh
Becoming magnetic
Speaking freely
Killing 99.9% of common sense
Eating a lot
Socializing
As you can see, surviving the virus was the easy part. Surviving those people was a totally different story. They did not only endanger their own lives, but also the lives of everyone around them. That was just some examples of our direct response to the pandemic.
Nonetheless, we're not here to shame COVIDIOTS or blame free-dumb fighters; we're here to inform and inspire with AWS IoT Core and M5Stack Core2 for AWS. Without any additional hardware, we will get mp3 files from reputable sources then play them on M5Stack Core2 for AWS to increase public awareness.
Knowledge is underrated to say the least, and we really don't have to make a business case for it. However, if there is only one life lesson in COVID-19 pandemic, we would argue it is "we need to know more". Aside from what we just covered, the virus spread so far and fast because for a while we didn't know how the virus transmits or even how to adequately test for it.
Let's learn together.
Game PlanOnce a day, AWS Lambda will run a Python script to get data from three different sources and save them in three AWS DynamoDB tables. This is needed to enforce consistency and to avoid overwhelming the device with large web pages. The contents of the tables can be accessed by the device (or anyone/anything else) using HTTPS GET request.
When we power on the device, it will connect to the last known Wi-Fi. If it is not available, the device will start it's own Wi-Fi and wait for us to connect and tell it which Wi-Fi to connect to. Once online, three mp3 sources will be presented to the user to choose from within three seconds. If no selection is made, the device will retrieve the last source used from device classic shadow. The device will request a random episode then it will play the mp3 file specified in the HTTPS response. While listening to the audio, we can retrieve a QR Code that will take us to the mp3 file web page if we need more details. We will have the ability to mute or skip the audio. Since knowledge is a two way street, we need to keep track of how the system is used. Not necessarily a direct feedback from the user, but an indication whether the user listened to the entire audio or skipped it.
Gathering Information (GI)Web Scraping is data scraping [process] used for extracting data from websites (wikipedia.org). We can't just take data without first understanding how to be good citizens of the internet. If that is not enough motivation, we may be blocked from certain websites if we maliciously increase their traffic by collecting their data. At the end of any base URL, add /robots.txt to find out what is allowed and what is disallowed.
GI - Centers for Disease Control and Prevention (CDC)To obtain good information, we go straight to the source. Getting data from CDC was the easiest by far since they have an Application Programming Interface (API) which is a connection between computers or between computer programs (wikipedia.org).
People would go to this human usable link https://tools.cdc.gov/medialibrary/index.aspx#/podcastseries/id/408549 to listen to the weekly podcast series that summarize content from the Morbidity and Mortality Weekly Report (MMWR) related to COVID-19. The main landing page for API documentation is https://open.cdc.gov/apis.html and we found Documentation of Content Syndication in that page. To learn about what's available through that API we go to Documentation at https://tools.cdc.gov/api/docs/info.aspx#intro
Long story short, https://tools.cdc.gov/api//v2/resources/mediatypes told us we have "Podcast Series" so we did https://tools.cdc.gov/api//v2/resources/media?mediatypes=Podcast%20Series. Found "id": 408549 which also appeared in the human usable link above, and URL for that is https://tools.cdc.gov/api/v2/resources/media/408549.json
Finally, we found what we came for in that json file:
"resourceUrl": "https://tools.cdc.gov/podcasts/media/mp3/323753_E_MMWR_Briefing_210531.mp3"Some good information can be found at https://www.cidrap.umn.edu/covid-19/podcasts-webinars. Since they don't have API like CDC did, we need to figure out how to obtain a list of mp3 files from that page. This is a static HTML page and code consistency is at the page administrator's discretion so long it looks OK to users. That became evident when we compared the link of the first episode with the link of the second episode.
Once we get past that, we still have to get URL for mp3 files by following the links which is a trivial exercise.
GI - University of Houston (UH)If you ever get tired of COVID propaganda and want to listen to the story of how our culture is formed by human creativity, head over to https://www.uh.edu/engines/. The University of Houston is not only my alma mater, they also have a radio program called "The Engines of Our Ingenuity". As a matter of fact, that program was the inspiration for this project. The reason we waited that long to talk about it is how hard it was to get data from the website. Unlike CDC, UH doesn't have API. We've seen how hard it is to get data out of CIDRAP's static pages. On top of that we found a missing link to episode 3107.
We also found that when users click on episode 2992 they are taken to 2991
All these inconsistencies had to be addressed by Lambda. On the upside, UH's mp3 files use HTTP; not HTTPS like CDC and CIDRAP. The problem here is, ESP8266Audio library does not play HTTPS mp3 URLs, but we will cross that bridge when we get there.
PlatformIOLearning PlatformIO has been on my radar for a while; all the nerds I know use PlatformIO.
Coming from Arduino IDE, learning how to perform tasks and use features has been very challenging. In theory, you can use the same project code in Arduino IDE and it should work once you install the proper libraries. The file platformio.ini contains the following libraries:
- m5stack/M5Core2@^0.0.4
For M5Stack Core2 for AWS - bblanchon/ArduinoJson@^6.18.3
To work with HTTPS responses, and MQTT Publish and Subscribe - 256dpi/MQTT@^2.5.0
Publish and Subscribe to MQTT topics - mrfaptastic/@^1.0.0
To avoid hard coding SSID and password and make M5Stack Core2 for AWS mobile - earlephilhower/ESP8266Audio@1.9.2
To play audio from a URL
Instead of uploading to GitHub a lot of files that I didn't even touch, I uploaded 4 files only. Three for PlatformIO and one for AWS. Here is how to use them:
Create a new blank project in PlaformIO.
Replace platformio.ini and main.cpp then add secrets.h in src folder.
When you upload this code for the first time, PlatformIO will download all the libraries specified in platformio.ini file. We will use SSID displayed on the screen as Thing name in AWS later. We will have to upload the code one more time at the end of this tutorial once we get more information from AWS to be added to secrets.h file.
In the previous section we mentioned that ESP8266Audio cannot play mp3 files over HTTPS. To some makers that would be the end of this project. Undeterred, we made couple of changes. We went to "<ProjectFolder>\.pio\libdeps\m5stack-core2\ESP8266Audio\src". In AudioFileSourceHTTPStream.h we added "WiFiClientSecure Sclient;" right after "WiFiClient client;". In AudioFileSourceHTTPStream.cpp we replaced "http.begin(client, url);" with "Sclient.setInsecure();" and "http.begin(url[4] == 's' ? Sclient : client, url);". The last line has been added to play both HTTP and HTTPS mp3 files.
ESP8266Audio library shouldn't be updated after download and modification.
Amazon Web Services (AWS)The amount of services can be overwhelming. To find a service quickly, we can just start typing the name in the search bar at the top.
We will use the following services:
- IAM
- DynamoDB
- IoT Core
- Lambda / EventBridge
- API Gateway
We need to create a role to be used by IoT Core Rule and execute Lambda function.
Type IAM in the search bar at the top.
Click Roles from the menu in the left.
Click Create role.
Click Next: Permissions.
Need to attache the following existing polices:
AmazonAPIGatewayInvokeFullAccess
AmazonDynamoDBFullAccess
CloudWatchLogsFullAccess
AWSIoTFullAccess
AWSLambda_FullAccess
AmazonEventBridgeFullAccess
Click Next: Tags.
Click Next: Review.
Give the Role a catchy name then click Create role.
We're not done with the Role yet, so click it.
Remember this Amazon Resource Name (ARN) arn:aws:iam::627541763378:role/DoItAll
Select Trust relationships tab and click Edit trust relationship.
Add more services as follow:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"dynamodb.amazonaws.com",
"apigateway.amazonaws.com",
"iot.amazonaws.com",
"lambda.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}Click Update Trust Policy.
We're done with IAM.
We need to create three tables to store data from our three websites. We also need to create one more table to record the outcome of playing mp3 files; Enlightened or Skipped.
Type DynamoDB in the search bar at the top.
Click Create table.
Populate Table name and make sure Partition key is a number.
Turn off Auto scaling for Read.
Turn off Auto scaling for Write.
Click Create table.
Create two more tables similar to this one with different names.
Create one more table slightly different.
We got all four tables.
We need to create a policy then a thing. We also need to create a rule.
Type IoT Core in the search bar at the top.
Expand Secure on the left.
We will start by creating a Policy.
We will give access to everything just as PoC, but security must be tightened before deployment.
We now have a policy to use.
Create a thing to use the Policy we just created.
We're creating one thing so click Next.
We'll need a Classic Shadow for the device. Notice that we used SSID displayed on the screen while waiting for Wi-Fi info.
We need to have the following as Device Shadow Statement:
{
"state": {
"desired": {
"TableNumber": 0
},
"reported": {
"TableNumber": -1
}
}
}Click Next.
Click Next to auto-generate a new certificate.
Use the Policy we just created.
We only need Device certificate, Private key, and Root CA 1 to add them to secrets.h file.
The page will not let you move forward until you download Public key as well.
We need a rule to send MQTT publishes to DynamoDB.
Give it a name.
Change SQL to "SELECT * FROM 'Tracking'" then click Add action.
Splitting into columns is a clever new option that will save us a lot of work. Let's use it.
Scroll to the bottom to Configure action.
Add destination table and role to be used.
Click Add action.
Click Create rule.
Rule has been created.
This will be the Python code that scrapes data and save it in DynamoDB tables. We also need to create a trigger to run this code once a day.
Type Lambda in the search bar at the top.
Click Create Function.
This is going to be Python from scratch.
Use the role we created earlier.
We now have almost blank function.
Replace the code with Python code from the link at the bottom of this tutorial.
Click Deploy.
Scroll up and click Add trigger.
We may wanna let it initially run at higher rate (like 10 minutes) to make sure tables are populated before we query them later. However, we don't wanna keep it running that frequent to avoid going over the free tier allowance. Don't ask me how I learned this.
Click Add.
Trigger has been created.
Click General configuration.
Click Edit to increase Memory and Timeout then click Save.
Finally, we need a way to access DynamoDB data. The easiest way is through HTTPS get request. The ultimate goal here is to create URL that looks like this:
Base URL/{collection}/{episodenumber}
We need that to replace {collection} with the table name and {episodenumber} with the record we are trying to retrieve.
Type API Gateway in the search bar at the top.
Scroll down.
Click Build from REST API box.
Click OK.
Select New API and give it a creative name.
We still don't have Base URL, but it's a good start.
Select Create a Resource from Actions at the root.
Don't forget to add curly brackets.
Select Create a Resource from Actions at the previous Resource.
Don't forget to add curly brackets.
Click on the API you just created.
We now have the path, we just need to decide what to do when we get there.
Select Create Method from Actions at the last Resource.
We need to access this method by GET.
This method in turn will access DynamoDB via POST and the Action will be GetItem.
Use ARN you obtained above when you created the Role.
The following is the request-response workflow where M5Stack Core2 for AWS is on the left and DynamoDB on the right. We need to edit the mapping in the top right and bottom right squares.
Click Integration Request.
Click Mapping Templates.
Click Add mapping template.
Type application/json.
Use the following text then click Save at the bottom.
{
"TableName": "$input.params('collection')",
"Key":{
"EpisodeNumber": {
"N": "$input.params('episodenumber')"
}
}
}Click Method Execution to get back to workflow.
Click Integration Response.
Click on the arrow.
Click Mapping Templates.
Click application/json.
Use the following text then click Save at the bottom.
{
"Title" : "$input.path('$').Item.Title.S",
"Audio" : "$input.path('$').Item.Audio.S",
"QRCode" : "$input.path('$').Item.QRCode.S",
"EpisodeNumber" : "$input.path('$').Item.EpisodeNumber.N"
}Click Method Execution to get back to workflow then click Test.
Fill out collection and episodenumber boxes.
Click Test.
Test worked internally, but will not work from outside until we deploy.
Select Deploy API from Actions.
Give this stage a name which will appear at the end of Base URL.
Click Invoke URL.
That's actually a good sign.
So, just add table and episode number.
That's it! Base URL (with slash at the end) goes to StageURL in secrets.h file. Speaking of which, the last value we need to insert in that file is AWS_IOT_ENDPOINT. That value can be found at:
https://console.aws.amazon.com/iot/home?region=us-east-1#/settings
Make sure you use your correct region.
ConclusionIt is ironic that my largest project to date has no hardware hacking. That has been possible because of the versatility and reliability of AWS IoT Core and M5Stack Core2 for AWS. Unlike some other cloud providers, when AWS gone through updates, none broke my project.
In the age of pseudo science and conspiracy theories, this project is much needed. I usually publish my projects before the contest is over, and if I don't publish this one I would be missing the point of my own project. I spent a lot more time documenting than building. I always avoid redoing documentations and tutorials, but made an exception and took the time to document each step to make this project accessible to everyone.
In a healthy spaces solution, a smart device empowers, and will never replace, an educated user.
Demo
Comments