The Things Industries(TTI) V3 Azure IoT Connector Revisited

Way back in November 2020 I published my The Things Network V2 Azure IoT Hubs & IoT Central Gateway. That project was about building a The Things Network(TTN)HTTP Application integration which enabled Azure IoT Hub and Azure IoT Central connectivity with Azure IoT hub Device Provisioning Service(DPS) provisioning support.

This project used a "buzzword compliant" selection of Microsoft Azure services but it didn't support cloud to device(C2D) messages, had message ordering issues, and was complex to deploy and setup. (there were other issues but it is not worth revisiting them here)

Then in March 2021 I tried again with my The Things Industries(TTI) V3 Azure IoT Connector which was a TTI (Message Queue Telemetry Transport) MQTT Integration that also used the TTI Application and End DeviceAPIs.

This version used MQTTNet(which is a great library) and the TTI devices connected to the Azure IoT Hub or Azure IoT Central as the application started. This process was slow even when I used multiple threads and paginated the Application and EndDevice requests.

The application was a lot easier to debug as I could run it on my desktop, and easier to configure as I shifted the configuration back to an appsettings.json file (I may revisit my decision to drop Azure Key Vault support).

This version had basic Azure Digital Twin Definition Language(DTDL) support so that devices could be "automagically" provisioned in Azure IoT Central.

I also added C2D support for both Azure IoT Hubs and Azure IoT Central with tracking of message delivery based on the downlink message payload confirmation flag. I found the ordering of TTI delivery progress updates could be problematic.

After using the MQTT based integration in a production environment I found it was too "statefull" and didn't recover well from unexpected events. (there were other issues but it is not worth revisiting them here)

Then in October 2021 I decided my "learning journey" wasn't finished and that I would build another TTI connector, which used Azure Storage Queues for both C2D and D2C messages.

After trialing the application I realised that message ordering and deployment complexity could be an issue (I had forgotten my TTN V2 gateway learnings) so I paused the project. (Though I do think this project could be useful for some integration projects)

At this point I reviewed what I had learnt from my multiple TTI integration projects and decided to try yet again using a The Things Stack(TTS) web hook integration.

My "The Things Industries(TTI) V3 connector revisited" project is an Identity Translation Cloud Gateway, which maps LoRaWAN EndDevices to Azure IoT Hub Devices.

The connector creates a DeviceClient for each LoRaWAN device and can use an Azure Device Connection string or the Azure Device Provisioning Service(DPS).

With all of my integrations TTI has been the single source of truth (SSOT) for device configuration as the number and complexity of LoRaWAN configuration settings would have made it a Pain In The Arse(PITA) to mange from other applications. (I also considered using TTSEndDevice templates for device creation which I may come back to)

A limitation of the current version is that an EndDevice will connect to an Azure IoT Hub (application configuration connection string or Azure IoT Hub DPS provided) and will process C2D messages only once a TTI uplink or Azure IoT Hub D2C message has been received by the integration.

This could be an issue (especially after the integration is restarted) or a new device configured. I have considered adding a couple of Azure HTTP trigger functions that applications call can to check a device's connection status and optionally initiate a connection. (In the short-term simulating an uplink from the TTI EndDevice user interface or API should work)

I started with D2C messaging, then added C2D messaging, then added Device Provisioning(DPS) with DTDLV2 support, then extended C2D messaging, then finally implemented Azure IoT Central D2C and C2D (with parameter less, single value, and JavaScript Object Notation(JSON) payload commands)

The core of the application is five Azure HTTP trigger functions (the sent function is currently unused) and a method which is called for C2D (wired up with SetReceiveMessageHandlerAsync method) messages.

An Azure IoT Hub can invoke methods(synchronous) or send messages(asynchronous) to a device for processing. The Azure IoT Hub DeviceClient has two methods SetMethodDefaultHandlerAsync and SetReceiveMessageHandlerAsync which enable the processing of direct methods and messages.

After some experimentation in previous TTI Connectors I found the synchronous nature of DirectMethods didn’t work well with LoRAWAN's often “irregular” uplinks so they are currently not supported.

The integration makes extensive use of Microsoft.Extensions.Logging functionality and Azure Application Insights so debugging, monitoring and fault finding is less time consuming.

I have added useful "meta data" to the individual log items so it is easier to track all of the steps executed to process an event e.g. the LockToken used in the ReceiveMessageCallback, AbandonAsync, CompleteAsync and RejectAsync C2D message processing.

Application Configuration Overview

The application can be configured with an appsettings.json file (useful for desktop development and debugging)

{
   "Values": {
      "AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=...",
      "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated",
      "APPINSIGHTS_INSTRUMENTATIONKEY": "..."
      },
      "TheThingsIndustries": {
      "WebhookBaseURL": "https://....eu1.cloud.thethings.industries/api/v3/as/applicat      ions",
      "Applications": {
         "seeeduinolorawan": {
         "webhookId": "azure-iot-hub-connector",
         "APIKey": "..."
         },
         "Wisnode Devices": {
         "webhookId": "azure-iot-hub-connector",
         "APIKey": "..."
         },
         "dragino-lht65": {
         "webhookId": "azure-iot-hub-connector",
         "APIKey": "..."
         },
         "SeeeduinoLoRaWAN100": {
         "webhookId": "azure-iot-hub-connector",
         "APIKey": "..."
         },
         "rak3172": {
            "webhookId": "azure-iot-hub-connector",
            "APIKey": "..."
         },
       "application1": {
          "webhookId": "azure-iot-hub-connector",
          "APIKey": "..."
        }
      }
   },
   "AzureIoT": {
      "DeviceClientCacheSlidingExpiration": "P2H30M",
      "IoTHub": {
         "IoTHubConnectionString": "HostName=...",
         "Applications": {
            "SeeeduinoLoRaWAN": {
            "DtdlModelId": "dtmi:ttnv3connectorclient:SeeeduinoLoRaWAN4cz;1"
            },
            "Wisnode Devices": {
            },
            "Dragino LHT65": {
            }
         }
      },
      "DeviceProvisioningService": {
         "IdScope": "0ne..",
         "Applications": {
         "seeeduinolorawan": {
            "DtdlModelId": "dtmi:ttnv3connectorclient:SeeeduinoLoRaWAN4cz;1",
            "GroupEnrollmentKey": "...",
         },
         "Wisnode Devices": {
            "GroupEnrollmentKey": "..."
         },
         "dragino-lht65": {
            "GroupEnrollmentKey": "..."
         },
         "rak3172": {
            "GroupEnrollmentKey": "..."
         },
      "application1": {
         "DtdlModelId": "dtmi:ttnv3connectorclient:FezduinoWisnodeV14x8;4",
         "GroupEnrollmentKey": "..."
      }
   }
},
"IoTCentral": {
   "methods": {
      "LightsGoOn": {
         "Port": 10,
         "Payload": "{\"value_1\": 1}"
      },
      "LightsGoOff": {
         "Port": 10,
         "Payload": "{\"value_1\": 0}"
      },
      "value_0": {
         "Port": 20
      },
      "value_1": {
         "Port": 21
      },
      "value_2": {
         "Port": 22
      },
      "TemperatureOOBAlertMinimumAndMaximum": {
         "Port": 23
      },
   }
}

The preferred approach for staging and production deployments)is to use the Azure PortalAzure function configuration blade

The Things Industries Webhook Integration Configuration

To send downlink and receive uplink messages a TTI Application and TTI Connector have to be provisioned and API Keys configured.

BEWARE – TTN URLs and Azure IoT Hub device identifiers are case sensitive

The TTI Connector requires the webhookbaseURL, then for each TTI application and an API Key, and WebhookId

When an Azure Functions is called the Azure Function Host Key is passed in an HTTP Header called "x-functions-key"

The downlink message processing is secured with a TTI App Key

When a TTI webhook downlink endpoint is called the TTI App Key is passed in the in standard HTTP Authorization header.

Azure IoT Hub Connectivity Configuration

The TTI Connector requires a Shared Access Signature(SAS) device policy connection string to connect to an Azure IoT Hub.

The Azure IoT Hub devices have to provisioned manually or via the Azure IoT Hub REST API. I have trialed an Azure Logic Application which manages device provisioning and can robustly handle the compensating transactions required when operations fail.

If both/neither of Azure IoT Hub/Azure IoT Hub Device provisioning(DPS) support is configured the TTI Connector application will not start.

BEWARE : the Azure IoT Hub device IDs are a case-sensitive string (up to 128 characters long) of ASCII 7-bit alphanumeric characters plus certain special characters: -. + % _ # * ? ! ( ), : = @ $ '

Azure IoT Hub Device Provisioning Service(DPS) Configuration

The TTI Connector supports the Azure IoT Hub Device Provisioning Service(DPS) for standalone Azure IoT Hub applications. The TTI Connector implementation also supports Azure IoT Central Digital Twin Definition Language (DTDL V2) for device configuration.

The Azure IoT Hub Device Provisioning Service supports device attestation with X.509 certificates, Trusted Platform Modules(TPM) and Symmetric keys using Shared Access Signature (SAS) Security tokens.

The Things Industries(TTI) V3 Azure IoT Connector only supports Symmetric Key device attestation.

If both/neither of Azure IoT Hub/Azure IoT Hub Device provisioning(DPS) support is configured the TTI Connector application will not start.

The Azure IoT Hub Device Provisioning Service(DPS) has a service-level setting that determines how devices are assigned. There are four supported allocation policies:

• Evenly weighted distribution: linked IoT hubs are equally likely to have devices provisioned to them. The default setting. If you are provisioning devices to only one IoT hub, you can keep this setting.
• Lowest latency: devices are provisioned to an IoT hub with the lowest latency to the device. If multiple linked IoT hubs would provide the same lowest latency, the provisioning service hashes devices across those hubs
• Static configuration via the enrollment list: specification of the desired IoT hub in the enrollment list takes priority over the service-level allocation policy.
• Custom (Use Azure Function): A custom allocation policy gives you more control over how devices are assigned to an IoT hub. This is accomplished by using custom code in an Azure Function to assign devices to an IoT hub. The device provisioning service calls your Azure Function code providing all relevant information about the device and the enrollment to your code. Your function code is executed and returns the IoT hub information used to provisioning the device.

In my test environment I use the evenly weighted distribution and when I provisioned 1000's of Devices they were spread across my five Azure IoT Hubs.

Azure IoT Central DPS Configuration

The TTI Connector supports the Azure IoT Hub Device Provisioning Service(DPS) which is required (there is a way of provisioning individual devices) for Azure IoT Central applications. The TTI Connector implementation also supports Azure IoT Central Digital Twin Definition Language (DTDL V2) for "automagic" device provisioning.

If both/neither of Azure IoT Hub/Azure IoT Hub Device provisioning(DPS) support is configured the TTI Connector application will not start.

The first step was to configure and Azure IoT Central enrollment group (ensure “Automatically connect devices in this group” is on for "zero touch" provisioning) and copy the IDScope and Group Enrollment key to the TTI Connector configuration

I then created an Azure IoT Central template for my RAK3172 breakout board based.Net Core powered test device.

The Device Template @Id can also be set for a TTI application using an optional dtdlmodelid which is specified the the TTI Connector configuration.

Azure IoT Hub Device to Cloud(D2C)

The LoRaWAN devices connect to an Azure IoT Hub with a Shared Access Signature(SAS) device policy connection string. I’m using Device Twin Explorer to display Telemetry from and send messages to my sensor nodes.

If the payload has been decoded by payload formatter, it is post processed then included in the message payload.

try
{
	JObject telemetryEvent = new JObject
	{
		{ "ApplicationID", applicationId },
		{ "DeviceEUI" , payload.EndDeviceIds.DeviceEui},
		{ "DeviceID", deviceId },
		{ "Port", port },
		{ "Simulated", payload.Simulated },
		{ "ReceivedAtUtc", payload.UplinkMessage.ReceivedAtUtc.ToString("s", CultureInfo.InvariantCulture) },
		{ "PayloadRaw", payload.UplinkMessage.PayloadRaw }
	};

	// If the payload has been decoded by payload formatter, put it in the message body.
	if (payload.UplinkMessage.PayloadDecoded != null)
	{
		EnumerateChildren(telemetryEvent, payload.UplinkMessage.PayloadDecoded);
	}

	// Send the message to Azure IoT Hub
	using (Message ioTHubmessage = new Message(Encoding.ASCII.GetBytes(JsonConvert.SerializeObject(telemetryEvent))))
	{
		// Ensure the displayed time is the acquired time rather than the uploaded time. 
		ioTHubmessage.Properties.Add("iothub-creation-time-utc", payload.UplinkMessage.ReceivedAtUtc.ToString("s", CultureInfo.InvariantCulture));
		ioTHubmessage.Properties.Add("ApplicationId", applicationId);
		ioTHubmessage.Properties.Add("DeviceEUI", payload.EndDeviceIds.DeviceEui);
		ioTHubmessage.Properties.Add("DeviceId", deviceId);
		ioTHubmessage.Properties.Add("port", port.ToString());
		ioTHubmessage.Properties.Add("Simulated", payload.Simulated.ToString());

		await deviceClient.SendEventAsync(ioTHubmessage);

		logger.LogInformation("Uplink-DeviceID:{deviceId} SendEventAsync success", deviceId);
	}
}
catch( Exception ex)
{
	logger.LogError(ex, "Uplink-DeviceID:{deviceId} SendEventAsync failure", deviceId);

	// If retries etc fail remove from the cache and it will get tried again on the next message
	_DeviceClients.Remove(deviceId);
}

Azure IoT Hub Cloud to Device(C2D)

Basic Azure IoT Hub C2D messaging only requires a port number, the TTI confirmation, queue and priority use default values if not provided.

• Confirmation – True/False
• Queue – Push/Replace
• Priority – Lowest/Low/BelowNormal/Normal/AboveNormal/High/Highest

These options are specified in message properties. For testing this functionality I use Azure Device Explorer Twin application which also display message delivery progress.

If the payload is not valid JSON is is assumed to be Base64 encoded (additional validation required) and copied into payload_raw field of the downlink message.

If the payload is valid JSON it is “grafted”(couldn’t think of a better word) into the TTI downlink message decoded_payload field.

Azure IoT Central Device to Cloud(D2C)

The connector "transforms" the output of The Things Industries(TTI) MyDevices Cayenne Low Power Protocol(LPP) payload formatter(It also supports custom encoders/decoders but this has not been extensively tested) so that it can be ingested by Azure IoT Central.

The Azure function for processing TTI Uplink messages first deserialises the JSON payload discarding any LoRaWAN control messages and messages with empty payloads.

To test more complex scenarios I created an Azure IoT Central Device Template which had a “capability type” of Location.

If the message has been successfully decoded by a payload formatter the PayloadDecoded contents will be “grafted” into the Azure IoT Central Telemetry message.

The Azure IoT Central Location Telemetry message has a slightly different format to the output of the TTI Cayenne LPP Payload formatter so the payload has to be “post processed” (With the new Azure IoT Central map telemetry on ingress functionality this may not be required).

I may have to extend the post processing to support other Cayenne LPP or third party payload formatters.

Azure IoT Central Cloud To Device(C2D)

To send a downlink message, TTI needs a LoRaWAN port number (plus optional queue, confirmed and priority values) which can’t be provided via the Azure IoT Central command setup so these values are configured in the Integration configuration.

My integration uses only offline queued commands as often messages won’t be delivered to the sensor node immediately, especially if the sensor node only sends a message every half hour/hour/day.

Each TTI application has zero or more Azure IoT Central command configurations which specify the LoRaWAN port number, plus optional payload, TTI downlink message confirmed, priority and queue settings.

Parameter Less Commands

Even though the command has no parameters the downlink message payload has to be configured(Currently only JSON encoded payloads, considering raw Base64 payload support)

This example shows how to configure commands for turning a light on and off using the built in Cayenne LPP payload formatter.

Enumeration Parameters

This example shows how to configure a command for turning a fan on and off by selecting the desired state from a list of options.

Single Value Parameters

This example shows how to configure a command for setting the minimum temperature for alerting.

JSON Parameters

This example shows how to configure a command for setting the minimum and maximum temperatures for alerting.

Delivery confirmations

For processing message delivery confirmations a correlation identifier containing the Message LockToken is added to the correlation_ids in downlink payload.

The only required message property is the LoRaWAN port number, confirmation, queue, priority and payload fields are optional.

If the port number property or any of others is incorrect DeviceClient.RejectAsync is called which deletes the message from the device queue and indicates to the server that the message could not be processed.

The message delivery confirmation process is tracked using an Azure Token which is stored in the TTI CorrelationID.

Unconfirmed Messages

The TTI Connector calls the CompleteAsync method (with the LockToken from the TTI CorrelationIDs list) which deletes the message from the Azure IoT Hub device queue when the "Queued" Azure function is called.

Confirmed Messages

If message delivery succeeds (Ack function called) the CompleteAsync method (with the LockToken from the TTI CorrelationIDs list) is called which removes the message from the Azure IoT Hub device queue.

If message delivery fails (Failed function called) the AbandonAsync method (with the LockToken from the TTI CorrelationIDs list) is called which puts the downlink message back onto the Azure IoT Hub device queue.

If message delivery is unsuccessful (Nack function called) the RejectAsync method (with the LockToken from the CorrelationIDs list) is called which deletes the message from the device queue and indicates to the server that the message could not be processed.

The way message Failed(AbandonAsync), Ack(CompleteAsync) and Nack(RejectAsync) are handled needs some more testing to confirm my understanding of the sequencing of TTI confirmed message delivery.

BEWARE

The use of Confirmed messaging with devices that send uplink messages irregularly can cause weird problems when the Azure IoT hub downlink message times out and is resent.

Executive Summary

A lot of effort has gone into this project it's been more than a year in the making. I have learnt a lot about LoRaWAN, and how The Things Industries works.

Sometimes it's stupid things like a typo that slows progress

I soak test the software for a month before I'm confident that it's ready for production but a couple of times I have hit my Azure Spending Limit which disabled all my services so I had to rerun the soak test.

If you have any questions or feedback message me here, I'm on Twitter and there is a lot more detail of my "learning journey" on my blog.