This document will describe building a Google Sheet which receives household current data from a Particle Photon. At the end of each 24 hour period, the data will be copied over to a new sheet, which will be analyzed by a script which compares current magnitude changes between samples against known appliance signatures to determine how much power was consumed by a given appliance. The total power usage for the day, as well as the power consumed by each appliance, is then copied over to a “DailyLog” sheet which tracks each day’s totals. This guide assumes you are using a Particle Photon (https://www.particle.io/) and have followed their instructions on initial setup, etc. You will also need access to a Google Drive and have some basic familiarity with spreadsheets.
Begin by going to your Drive and creating a new Google Sheet that will collect the data from your Particle Photon. Give your spreadsheet a name, anything you like. The first column header must be called “Timestamp” (case sensitive). Subsequent columns can be given any name, but for the purpose of this project we will fill them out as:
Row 1 must match the JSON keys described later on in this document exactly. Row 2 is informational so the user can see which frequencies are represented in each column. This is configurable by the user and will be described in the Photon code portion of this document.
Row 3 will contain some formulas which help track total power consumption. Cell A3 should contain the formula:
=sum(-1*A4,INDEX(A4:A, COUNT(A4:A)))*24
This formula subtracts the first timestamp from the last timestamp and multiplies by 24 to give the total amount of time in hours from the first sample to the last. Cells B3 and M3 should respectively contain the formulas:
=average(B4:B)*A3*120/1000
=average(M4:M)*A3*120/1000
These formulas find the average current, multiply by the number of hours (A3), multiply by 120V and then divide by 1000 to get the total number of kilowatt-hours consumed on each phase for the day.
Highlight column A and click Format>Number>Date time in order to show the date and the time on the timestamp. Then, highlight cell A3 and change its format to Number>Number, since it is a number of hours rather than a timestamp.
You should end up with something like this:
Make a note of the name of the particular sheet within the document (“Sheet1” by default), as you will need it for the script later on. The Photon will be sending data to this spreadsheet, adding a new row each time. Add an additional sheet within this spreadsheet and insert this function in the first cell(A1):
=SORT(QUERY(Sheet1!A:Z,"order by A desc limit 90"),1,1)
This will copy the last 90 entries from Sheet1 and continuously update them, which will allow us to create dynamically updating charts. You can create additional sheets changing 90 to whatever value you want, allowing you to create charts of different time periods.
Like this:
Google Sheets NOTE: sometimes, when you change a cell in Sheets, it doesn’t appear to take effect until you do something in another cell. If nothing happens when you add this “sort” function, type a random character into a random cell and hit enter. (Don’t forget to delete it later.) Or, just continue with this how-to and trust that it’s about to pop in there.
The Google Sheets ScriptIn order to set the spreadsheet up to receive data from the Photon, we’re going to create the script described here. From Google Sheets click Tools>Script Editor. This will open the script editor in a new tab. Give your project any name you like. Then replace the default text inside the Code.gs with the Google Sheets JavaScript(attached below).
You should have ended up with something like this:
Note that in line two, the variable SHEET_NAME must coincide with the name of the particular sheet within your spreadsheet that is receiving the data. So if you change it from the default of “Sheet1”, be sure to change it here as well.
Click the save icon, then run the setup function by clicking Run>Run function>setup. You will get a popup window to review permissions. Click “Review Permissions”, then in the next window click on your Google account. After that, you may receive a prompt that says “This app isn’t verified, ''if so, click “Advanced”, and then “Go to GenericScriptName(unsafe).” Then you can click “Allow” to give the script permission to edit your spreadsheet.
While still in the script editor, click File>Manage versions to create an initial version of the script. Give it a description, such as “InitialVersion” and click “save new version” and then “ok”.
Next we need to set up triggers for these scripts. To do this, click Edit>Current project’s triggers, which will open the G Suite Developer Hub in a new tab. Click the “+ Add Trigger” button in the bottom right corner” and in the popup window, from the dropdowns select ''doPost”, “Head”, ''From Spreadsheet”, and “On form submit”, and then click save. The failure notification settings can be set according to your preference of how often you want to be notified when the script fails. There will probably be another popup window for you to select your Google account again, then ask permissions to allow the script to edit the sheet, click “Allow”.
You will also need to create a trigger to run the copy_delete function daily, which duplicates Sheet1, renames the duplicated sheet with the day’s date, and then deletes all of the rows on Sheet1 to start a new day’s data. Click “+ Add Trigger” once again and select “Copy_delete”, “Head”, “Time-driven”, “day timer”, and “10pm to 11pm” (or whatever your preference for time) from the dropdown options. Note that the script will rename sheet with the day’s date, so you’ll want the time to be before midnight. Once again, the failure notifications can be set to your preference.
Likewise, add triggers for the appliance_id and getDailyTotal functions. These scripts should run AFTER the copy_delete function has run, so if you selected “10pm to 11pm” you could select “midnight to 1am” for appliance_id and “1am to 2am” for getDailyTotal, which must be run last.
You should end up with something like this:
Return to your script (the script editor tab) and click Publish>Deploy as web app. From the dropdowns select the relevant project version (probably 1), execute app as “Me”, and who has access “Anyone, even anonymous.” Click “deploy” and you will be given a url, which you will need for your webhook. Copy it and click “ok.” (You can save the link in notepad or something similar.)
For the webhook, we’re going to be elaborating on the method described in this tutorial. From the Particle Console, go to the Integrations page, and click on the “+ NEW INTEGRATION” box, then click “Webhook.” Give the webhook an Event Name, which will be the Particle.publishevent that initiates the webhook. For our purposes, it will be called “GenericWebhookName” (case sensitive). For url, paste the address that you were given when you published your script as a web app. Request Type is “POST”, Request Format is “Web Form”, and Device can be left as “Any.” The small dropdown at the bottom is “Advanced Settings” -
Click the “Advanced Settings” menu at the bottom and toggle the “Custom” radio button. Begin filling out the form fields.The order of the rows isn’t particularly important here. What’s worth noting is that the left column correlates with the column headers on your spreadsheet, and the right column is the key from the JSON key/value pairs you’re going to be sending. I’ve given them the same names for simplicity. All the rows are not shown in the pic below, but continue filling it out until you have a row for every column in your spreadsheet except the “Timestamp.”
Everything else can be left at it’s default setting. Click “Create Webhook.” The complete webhook should look like this:
Start off by updating your photon to the latest version of firmware. Click on the crosshairs(devices) on the bottom left of the build.particle.iowebpage for your devices. Click the drop down for the device you want to use and select the most recent system firmware. 0.8.0-rc.14 or newer is recommended, although older rc’s may work. The selected firmware target will be downloaded to your device the next time it is flashed.
For our project, you can use this code to upload to your Photon. For the general case, however, the format of publishing data to a spreadsheet using this webhook will look something like this:
Particle.publish("GenericWebhookName","{\"IrmsA\":\"" + tIrmsA + "\",\"A1\":\"" + A_1 + "\",\"A2\":\"" + A_2 + "\",\"A3\":\"" + A_3 + "\",\"A4\":\"" + A_4 + "\",\"A5\":\"" + A_5 + "\",\"A6\":\"" + A_6 + "\",\"A7\":\"" + A_7 + "\",\"A8\":\"" + A_8 + "\",\"A9\":\"" + A_9 + "\",\"A10\":\"" + A_10 + "\",\"IrmsB\":\"" + tIrmsB + "\",\"B1\":\"" + B_1 + "\",\"B2\":\"" + B_2 + "\",\"B3\":\"" + B_3 + "\",\"B4\":\"" + B_4 + "\",\"B5\":\"" + B_5 + "\",\"B6\":\"" + B_6 + "\",\"B7\":\"" + B_7 + "\",\"B8\":\"" + B_8 + "\",\"B9\":\"" + B_9 + "\",\"B10\":\"" + B_10 + "\"}",5,PRIVATE);
There are a few things to keep in mind with regard to the constants defined in the Photon firmware below. The number of samples must be a power of 2, and the sampling frequency must be an integer multiple of the number of samples. We’ve found that 4096 seems to be about the max sample rate that the Photon can reliably achieve. The constant “harm” is the number of harmonics to be captured on each phase, and “base_harm” is the lowest one. For example, we’ve defined “base_harm” as 20 and there are 10 harmonics per phase (20-200 in multiples of 20). Our testing has shown this to be roughly the range of useful information for our project.
Verify and flash your firmware (download the code to your device). This command is publishing an event called “GenericWebhookName” and generating a string of JSON formatted text that is sent to our script we created earlier. The variables being appended to the string above are string type. If all goes well, in your Particle console, you should see something like this:
Go back to you spreadsheet now and you should start seeing rows being added with your data. On Sheet2, which you set up to show the last X entries from Sheet1, you can create charts by selecting columns, and the charts will continuously update with your data.
Additional Sheet SetupNow that your sheet is receiving data, you’ll need to set up a sheet which holds the individual appliance signatures that you will be testing against. Create a new sheet and give it the name “Appliances” (needs this exact name for the appliance_id script to work). Row 1 should be headers used to identify the columns.
To obtain the appliance signature, turn the appliance on and note on Sheet1 where the current changes. You may need to turn the same appliance on and off several times, noting the time, so you can find it’s signature. Single phase appliances (like your microwave) will only show up under the “A” phase or the “B” phase. Appliances like your air conditioner will change across the whole row. Once you “see” your appliance turn on as you’re watching Sheet1:
Highlight and copy the row immediately before and after the change you noticed:
Next, return to the Appliances sheet and paste the data into some blank cells:
Insert a function below this data that calculates the difference:
And copy that formula under each column. Once you have the entire row finished highlight and paste it into the next empty row in the list. Paste using “Ctrl+shift+V” in order to paste the values only and not the formula. This difference is your appliance’s signature.
Once you’ve copied the signature data into the appropriate row, you can delete the extra data and fill in the appliance name. You can also place 0’s in each column that is blank. The appliance_id script ignores frequencies where the magnitude change is less than 1 anyway (because the signal is drowned out by the noise), so you can replace any numbers less than 1 with 0’s for the sake of neatness. When you’re finished, it should look something like this:
Next, you’ll need a sheet to log the daily power usage. Create a new sheet titled “DailyLog”. For row 1, columns A-E should be labeled “Date”, “Phase A”, “Phase B”, “Total Power”, and “Total Cost”, with the subsequent columns matching your individual appliance names in the same order they are listed on your Appliances sheet. It will look something like this:
You should now have a Google Sheet document which is receiving real time data from your Particle Photon into Sheet1. Once per day at the times you’ve designated, the copy_delete script should run first, then the appliance_id script, and finally the getDailyTotal script. To start the project, because we are keeping seven days of data (if you followed this tutorial exactly), the getDailyTotal script is going to delete the seventh sheet (or tab) from the left. Because you don’t want the script to accidentally delete your “DailyLog” or “Appliances” sheets, you will need to add seven blank sheets to the document. The script is counting sheets from the left and deleting the seventh sheet by position, so move the sheets you want to keep all the way to the right. Something kind of like this:
The copy_delete trigger, that sheet will be duplicated and renamed with the day’s date, and Sheet1 will be wiped clean to start a new day.
The appliance_id script will then comb through that day’s data and compare the magnitude changes in each row to each of the appliance signatures on the Appliances sheet. It will create one new column for each appliance. The script basically begins by comparing the current change only. If the current change is within 2 amps of the current change on the appliance’s signature, it then goes on to compare each frequency which has a magnitude change greater than 1 on that appliance’s signature. If they are within the specified tolerance of each other (this can be changed in the variable “tolerance” on the appliance_id script (Line 104 in the Google script)), then the script will insert the power consumed (in kWh) for that particular sample in that row under that appliance’s column. Once it has been designated as “on”, it simply looks for the current to drop by the amount in the appliance’s signature to determine it is “off”. Total power consumed by each appliance is accumulated and placed in row 3 of that appliance’s column. After the script has run, that day’s data will look something like this:
Looking at row 3, we have the total number of hours in column A, power consumed in kWh for phases A and B in columns B&M, and the power consumed by each appliance in columns from X onward.
Finally, the getDailyTotal script will run and will copy these totals over to our DailyLog page. It should look something like this:
There are a couple of variables in the getDailyTotal script that you’ll want to consider. The first is the cost variable, which you’ll need to define as the cost/kWh of power from your provider. The second is the days2keep variable, which is the number of days of data you want to keep. Google Sheets allows a maximum of 5 million cells in a document (Google Drive Support), so you won’t be able to store them indefinitely. This is given a value of 7 in the code above to store 7 days.
As you come up with new and creative ways to make calculations, move data more efficiently, or create charts and graphs to impress your friends, please share your ideas! (Don’t forget to move those new sheets to the right!)
Comments