Datacenter Monitoring

Downtime in a normal datacenter is rare, as the large budgets allow for redundant redundancy. From extra hardware to backup generators and multiple links to the Internet, monitoring of these centers is covered. For small datacenter providers, Rural ISPs, radio stations, or even small hospitals, this type of redundancy is cost prohibitive. Even so, personnel have the requirement to constantly monitor systems, HVAC performance, electrical continuity, etc.

Project Overview

The Information Technology staff is generally responsible for monitoring outages and when one occurs, someone has to drive to the site to see exactly what is going on (or off, as the case may be).

Using the kit, data collected from the monitor is sent via AT&T to PubNub, where the data can be displayed, analyzed and sent to a dashboard. With this data informed decisions can be made.

In the event of a failure, the data will continue to flow via AT&T providing information about the type of failure, the environmental conditions at the site, and potentially images of the site. Staff would then be able to quickly make a decision as to whether to call the fire department, electric company, or the provider of data services before heading to the site. 

From hardware to cloud service integration, this project requires many parts. This project focuses on integration of all of them, and leans heavily on others who have used and programmed hardware and software.

The following getting started comes mostly from this page: Snowmelt System Monitor.  Check out his page for a ton of great information regarding this very hardware.

Getting Started with the AT&T IOT IoT Kit

The best way to get started with your board is to follow Avnet Maker Faire Labs at: http://cloudconnectkits.org/system/files/MasterMakerFaire.pdf

This is a straight forward way to get your K64F and cellular shield up and running. If you follow the guide, note that you don't actually do anything with the hardware until you get to page 28. And when you do get to the hardware, I highly recommend you upgrade the firmware. There are actually two firmware updates, one for the K64F, and one for the Cellular Shield. Both boards should be updated before connecting them together, so update the K64F, (page 32, https://developer.mbed.org/handbook/FirmwareFRDM-K64F) and then update the firmware on your cellular shield. Visit https://starterkit.att.com/tutorials/cellular-shield-firmware-upgrade and follow the steps very closely.

Yes, everyone is making fun -- run the installer 3 times.

A couple notes:

Windows users, you may need to install mbed serial driver from https://developer.mbed.org/handbook/Windows-serial-configuration

Before I did the firmware update, I got stuck in bootloader mode. You will know if you have this issue if you see "bootloader" attach to yor PC instead of "mbed " as a disk attachment.  I also couldn't access the serial terminal data on Windows 10. Still can't. I can on a Windows 7 laptop, but my Windows 10 box doesn't have proper drivers and I have not taken the time to figure out why (because I have Windows 7 and that works, so..) 

Connecting AT&T Cloud Services

Now that you have your board up and talking to the net, let's get it online with the systems we need. AT&T offers two great tools - M2X and Flow.

AT&T M2X provides time-series data storage, device management, message brokering, event triggering, alarming, geo-fencing, and data visualization for your industrial Internet of Things (IOT) products and services.

Flow Designer is a robust web-based development environment where data driven applications can be designed and deployed with ease.

Check out this Avnet guide at: http://cloudconnectkits.org/system/files/GettingStartedGuide_ATT_IOT_rv1-3.pdf

This is a bit redundant, but at least look it over. If you have time to go through it, it will certainly help you get more familiar with the tools.

Once you are done with this, you will have:

The Starter Kit Flow, that you can edit to do your bidding

M2X collecting Data

A nice Avnet_ATT_Cellular_IOT project in your mbed IDE that you can change for the better.

Connecting to PubNub

PubNub is a message handling system that allows you to move your data from one application to another. From one to many users. This project uses PubNub as a tool for graphing data.

Create an account at pubnub.com

Create a New App by entering a name and pressing the big green button

Note here that you have a Demo Keyset. We can use that for starters.

Now, move over to Flow, and access your Starter Kit code.

We are going to take the data flow that we already are sending to M2X, and send it to Pubnub. To do this, we are going to use the import function in Flow.

Click on the "hamburger " icon in the upper right corner. It looks like 3 lines. Then choose import, Clipboard. This will allow you to import the following text:

[{"id":"33ddf60a.d8026a","type":"pubnub-keys","z":"828b2334.7d74e","pub_key":"pub-KEY","sub_key":"sub-KEY"},{"id":"611c3260.7f625c","type":"pubnub out","z":"828b2334.7d74e","keys":"33ddf60a.d8026a","channel":"SnowMelt","x":819.328125,"y":549,"wires":[]}]

This will create a new flow node. Place it next to the Heat Index node, and then connect it up, like this:

Building outbound to PubNub

Go ahead and click on this new node and set the keys. Those are over on your Pubnub page.

Click over to Pubnub, and select your project. Select your keys, and then click on the Debug Console on the left side. You should see a console window showing that it is now receiving data from Flow:

Note something interesting here (besides my extra data fields): My data is wrapped in an "eon " identifier. This is an added data node that allows us to use EON graphing. More on that later. But for now, since we are working in Flow, let's go ahead and modify the data stream to add this "eon ", so we will be set for graphs later.

Back to flow.

On the left side search, type "function ", and drag one onto the work surface, between the Heat Index and PubNub node.

Click on it, and add this code:

// if you want to use EON graphing,   
// you need to wrap the payload with "eon"  
//  
msg.payload = {  
"eon": {  
"temp": msg.payload.temp,  
"humidity": msg.payload.humidity,  
"heatIndex": msg.payload.heatIndex  
}  
};  
return msg;  

Talk about an easy function. We are just adjusting the msg.payload that came into the bubble to go out as something else.

Now the data flowing out of the Heat Index is rewritten before posting to PubNub.

Adding Sensors

Let's add the analog light sensor to our system. Go ahead and wire up your light sensor as follows:

Sensor GND to GND

Sensor VCC to 3V3

Sensor OUT to A0

The cellular shield headers carry the OUTER pins through. I think you need to stay away from A4, A5, D14 and D15, as they are connected to devices on the cellular shield.

If you look into the code you will see DigitalOut definitions that identify D1, D2, D6, D8, D9 and D10 in use by the modem. Stay away from those pins as well!

Great. Let's update our K64F Code to read the sensor. In mbed , go to your sensors.cpp file. search for the line "} //Read_HTS221() " about 422 lines down. Add the following code.

// Light Sensor  
AnalogIn light(A0);      
void Read_Light()  
{  
sprintf(SENSOR_DATA.Light, "%0.2f", light.read());  
printf("light in = %s\n\r", SENSOR_DATA.Light);  
}  

While we're here, you need to add a call to this new subroutine in the subroutine that calls it - read_sensors() , at the bottom of sensors.cpp. Add your new routine to the read_sensors subroutine, like this:

void read_sensors(void)  
{  
Read_HTS221();  
Read_Si7020();  
Read_Si1145();  
Read_motion_sensor();  
Read_GPS();  
Read_Light();  
} //read_sensors  

We are going to read the light level on A0 and post it to the SENSOR_DATA string. The string needs to be modified to have light as well. Where's that? Glad you asked. Go to Sensors.h and scroll down to the bottom. Add a line right after "GPS_Course ":

char Light[SENSOR_FIELD_LEN_LIMIT];

We also need to add light to our send string. In Config_me.h , Search for your iSensorsToReport , and add a new type. Mine's snowmelt :

#define TEMP_HUMIDITY_ONLY                                      1  
#define TEMP_HUMIDITY_ACCELEROMETER                             2  
#define TEMP_HUMIDITY_ACCELEROMETER_GPS                         3  
#define TEMP_HUMIDITY_ACCELEROMETER_PMODSENSORS                 4  
#define TEMP_HUMIDITY_ACCELEROMETER_PMODSENSORS_VIRTUALSENSORS  5  
#define SNOWMELT                                                6  
static int iSensorsToReport = SNOWMELT; //modify this to change your selection  

Now head over to main.ccp , and add a new case statement in the iSensorsToReport that looks like this:

case SNOWMELT:  
    // Modified for pete's sensor array  
    {  
        sprintf(modem_string, "GET %s%s?serial=%s&temp=%s&humidity=%s&light=%s %s%s\r\n\r\n", FLOW_BASE_URL, FLOW_INPUT_NAME, FLOW_DEVICE_NAME, SENSOR_DATA.Temperature, SENSOR_DATA.Humidity, SENSOR_DATA.Light, FLOW_URL_TYPE, MY_SERVER_URL);  
        break;  
    }  

This is actually the URL that is sent to feed Flow. We have added "&light=%s " to the string, and then "SENSOR_DATA.Light ". Run your program, and watch your serial USB output. You should see:

light in = 0.52  
Sending to modem : GET /DEMO/in/flow/climate?serial=snowmelt001&temp=91.86&humidity=24&light=0.52  HTTP/1.1  
Host: run-east.att.io  
JSON : {"status":"accepted"}  

We're not done yet. We need to add Light into our Flow! Head back to flow, and add Light to the list of data nodes:

"light": msg.payload.light,

Do the same for the M2X Lead-in function. I bet you can figure out the code. Check your PubNub quick. Look at that, you have Light in your data stream now. Nice.

Let's head over to M2X quick, and check it out there. Click on your device, and add a stream. Caps are important here. Make sure light is light, and not Light. Awesome. You are collecting your light data at M2X:

Let there be Light

WORK IN PROGRESS -- THE FOLLOWING SECTIONS ARE NOT UPDATED.  THE HARDWARE THAT I NEED TO FINISH MY PROJECT HAS NOT ALL BEEN TESTED, SO THE FOLLOWING MAY OR MAY NOT APPLY. 

Additional sensors for power, generator status (if a generator is present), HVAC system status (if available), motion detection, smoke detection will be added to the project. 

PubNub EON Graphs

We can add new data to our system. Let's make some charts! PubNub EON reads your channels and plots them in various ways. First go ahead and browse the PubNub EON site at: https://www.pubnub.com/developers/eon/

PubNub Eon

On a webserver, create a file monitor.html and paste code you see in the code section below. Adjust the keys and variables to suit your data stream. Done. If you are a web developer, you can do some magic and make your dashboard look pretty good.

Now let's create the Pubnub Block. Visit https://www.pubnub.com/blocks-catalog/twitter/ and click Try it Out. Chose your app and keyset, and import the block. You will be presented with the code. Simply edit your keys into it.

Click Start in the upper right, and then publish on the test payload. You should see a tweet show up from your twitter account in your twitter stream. You can also open another window and check out the debug by subscribing to twitter: input. Send another test tweet (probably with different text) and see the results:

Okay, let's get Flow to send us tweets when it's cold. Access your flow, and import the following code:

[{"id":"33ddf60a.d8026a","type":"pubnub-keys","z":"828b2334.7d74e","pub_key":"pub-KEY","sub_key":"sub-KEY"},{"id":"3850ec25.6d7374","type":"pubnub out","z":"828b2334.7d74e","keys":"33ddf60a.d8026a","channel":"twitter-input","x":1170.875,"y":665,"wires":[]},{"id":"81831022.4d9d2","type":"function","z":"828b2334.7d74e","name":"Check for Freezing Temperature","func":"// if temperature is <= 32 send a direct message tweet to a user\n\nuser = \"USERNAME\"; // change to your user name\n\nvar freezing = context.get('freezing')||0; // Get current freezing status\n\ntemp = msg.payload.probe0;\n\nif (temp <= 32) { // If it's freezing\n if (freezing===0) { // And wasn't freezing before\n msg.payload = {\"tweet\": \"DM \" + user + \" Freezing at \"+ msg.payload.probe0 + \"F\"};\n freezing = 1;\n }\n else freezing = 1; // It's freezing, or still freezing\n}\n\ncontext.set('freezing',freezing); // remember for next time\nmsg.freezing = freezing;\nreturn msg;","outputs":1,"noerr":0,"dependencies":[],"x":740.875,"y":592,"wires":[["88bbe03.6f76f2","3850ec25.6d7374","52e365e5.7871dc"]]},{"id":"88bbe03.6f76f2","type":"debug","z":"828b2334.7d74e","name":"","active":true,"console":"false","complete":"false","x":1077.59375,"y":552,"wires":[]},{"id":"52e365e5.7871dc","type":"debug","z":"828b2334.7d74e","name":"Freezing","active":true,"console":"false","complete":"freezing","x":1077.609375,"y":600,"wires":[]}]

Set your keys, and connect that up to the back of your Heat-Index node.

We will also add an overheating tweet, in case the temperature rises too high in the data center.  We will likely change this to a TXT message or email message to protect the privacy of the data center. 

Tips and Tricks

We learned a lot in developing this project. Here's some great hints for you:

The folks in #att and #development at the HacksterLive slack channel are your friends. They are great at helping out. Check them out: http://slack.hackster.io/

The mbed online development system is really nice. Be sure to look at example code. It is easy to import the code to your IDE, and start messing with it.

Depending on your project, you might want to consider adjusting your polling time. In the Config_me.h , change your interval as needed. Save data, save $.

// This constant defines how often sensors are read and sent up to FLOW  
#define SENSOR_UPDATE_INTERVAL_MS           5000; //5 seconds  
// #define SENSOR_UPDATE_INTERVAL_MS       60000; //1 minute  
// #define SENSOR_UPDATE_INTERVAL_MS       1800000; //30 minute  
// #define SENSOR_UPDATE_INTERVAL_MS       3600000; //1 hour  

When you reboot the K64F, you will see the board LED go white (booting), then red (Connecting to cell system), then green (transmitting). This is the normal operation.

Conclusion

The combination of the AT&T starter kit, AT&T cloud services and PubNub is an extremely powerful way to get your IOT project online. This project only scratches the surface of what is possible with these systems.