Cluster of Raspberry Pis in K8s to Teach Azure IoT Central

Introduction

This story begins last month when I accept a new class to teach Internet of Thing to specialization class of the User Experience Design course at the University of the Region of Joinville UNIVILLE. Brazil is facing the Covid-19 pandemic, and we can't do the classroom lessons in the university. Internet of Things is a very practical class, using Arduino's and Raspberry Pi's, and we normaly develop a End-to-End solution using Microsoft Azure IoT services. As a solution proposal for this situation, I had the idea of creating a cluster of Raspberry Pi so that students could access the Node-RED environment using the DHT11 sensor and build a solution connected to Azure IoT Central. The class has nineteen students, so I would need nineteen instances of Node-RED, in addition to the bundle of nodes to connect to Azure IoT Central.

Building the Cluster

I had already tried to create a Kubernets cluster with my Raspberry Pi's, without much success. For this reason I bought several Raspberry Pi's used on e-bay, totaling today more than 12 devices of different versions. For the cluster I selected the best ones, the Master node is a Raspberry Pi version 2, the main worker is a Raspberry Pi version 4 with 4GB of RAM (which I won in the Hacksters.io competition, thanks), and the other workers are divided between version 2, version 3 and even version 1. I decided to use the Kubernetes cluster so that I could distribute the workload across several Raspberry Pi's, and be able to run Node-RED in the real environment connected to the real DHT11 sensor. To configure the cluster I used the excellent video published by Jeff Geerling (https://www.youtube.com/watch?v=N4bfNefjBSw) that explains step by step how to use Ansible to install K3S (https://k3s.io/), which is a lightweight version of Kubernetes capable of running me Raspberry Pi's. Observing the behavior of the cluster, some Raspberry Pi's, mostly from version 1, are unable to run K3S and other services due to the lack of RAM and because I am using SD cards of different classes: from 4 to 10.

It was necessary to remove a Raspberry Pi version 3 due to overheating problems, and one of the Raspberry Pi version 1 stopped responding to the cluster, it probably must have corrupted the operating system.

Docker Image

As I mentioned, the nineteen students should have their individual instance of Node-RED, even though they are actually sharing the same Raspberry Pi. For this I created using another Raspberry Pi an image that is available on my Docker Hub (https://hub.docker.com/repository/docker/waltercoan/univillenodered) in this image the libraries were installed to access Azure IoT Central, the DHT11 sensor, in addition to the library for building Node-RED Dashboards.

Kubernetes + Docker + Access to GPIO

In order for the DHT11 sensor library to run, it needs to use the C library for Broadcom BCM 2835 as used in Raspberry Pi. This binary was compiled and pre-installed on the Docker image used in the solution. But even so, it was necessary to configure the Kubernetes Deployment so that the POD instance had full access to the address /dev/gpiomem in addition to requiring that the container be run with super user permissions. This Deployment configuration was repeated nineteen times to create an instance for each user.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nodered-deployment-1
spec:
  selector:
    matchLabels:
      app: nodered-1
  replicas: 1
  template:
    metadata:
      labels:
        app: nodered-1
    spec:
      securityContext:
        runAsUser: 0
        fsGroup: 0
      containers:
      - name: nodered-ctr
        image: waltercoan/univillenodered
        command: ["/bin/sh","-c"]
        args: ["npm start --cache /data/.npm -- --userDir /data"]
        resources:
          limits:
            memory: "128Mi"
        ports:
        - containerPort: 1880
        - containerPort: 1900
        - containerPort: 80
        securityContext:
          runAsUser: 0
          privileged: true
        volumeMounts:
        - mountPath: /dev/gpiomem
          name: gpiomem
      volumes:
      - name: gpiomem
        hostPath:
          path: /dev/gpiomem

Providing Internet access to Node-RED to students

The next challenge for this project was how to provide access to each instance of Node-RED for each student, and also ensuring that my provider's public IP was not fully exposed. For that, I used the article by Abhishek Gupta (https://medium.com/@abhishek1987/expose-kubernetes-services-with-ngrok-65280142dab4) that shows how to use the Ngrok proxy to redirect packages into the cluster. Basically for each Pod a ClusterIP Service was created that exposes the Node-RED Pod port to a new port accessible on the Kubernetes network. Then two new Deployments were created to run the Ngrok image for Raspberry Pi on two Pods (https://hub.docker.com/r/wernight/ngrok), it was necessary to use the paid version to support the number of tunnels needed.

apiVersion: v1
kind: Service
metadata:
  name: nodered-deployment-1
  labels:
    name: nodered-deployment-1
spec:
  type: ClusterIP
  ports:
    - port: 30001
      targetPort: 1880
      name: http
  selector:
    app: nodered-1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ngrok-deployment-1
spec:
  selector:
    matchLabels:
      app: ngrok-1
  replicas: 1
  template:
    metadata:
      labels:
        app: ngrok-1
    spec:
      securityContext:
        runAsUser: 0
        fsGroup: 0
      containers:
      - name: ngrok-ctr
        image: wernight/ngrok:armhf
        command: ["/bin/sh","-c"]
        args: ["ngrok authtoken NGROK_TOKEN; ngrok http nodered-deployment-1:30001"]
        ports:
        - containerPort: 4040
        securityContext:
          runAsUser: 0
          privileged: true

Eclipse Mosquitto

An Eclipse Mosquitto Deployment was also created to allow students to have the experience of creating an IoT solution using an MQTT broker.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mosquitto-deployment-1
spec:
  selector:
    matchLabels:
      app: mosquitto-1
  replicas: 1
  template:
    metadata:
      labels:
        app: mosquitto-1
    spec:
      containers:
      - name: mosquitto-ctr
        image: eclipse-mosquitto
        resources:
          limits:
            memory: "128Mi"
        ports:
        - containerPort: 1883
---
apiVersion: v1
kind: Service
metadata:
  name: mosquitto-deployment-1
  labels:
    name: mosquitto-deployment-1
spec:
  type: ClusterIP
  ports:
  - port: 31883
    targetPort: 1883
    name: http
  selector:
    app: mosquitto-1

Running the Cluster

As shown in the images below, the cluster supports the number of instances needed quite satisfactorily, even though some participating nodes are not able to run new pods.

End-to-End Solution

During the class, each student received the Ngrok URL for their Node-RED instance, and was instructed to create a logic that uses the DHT11 sensor to collect Temperature and Humidity information and send it to Azure IoT Central. The first step was to configure a flow with an event injection node and the node for reading the DHT11 sensor values.

The message sent by the stream needed to be modified to the standard required by the Azure Device node, as shown in the following image.

[{"id":"4521535f.dee1dc","type":"inject","z":"a56382b8.978a5","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"1","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":526,"y":125,"wires":[["4569dcca.c9c384"]]},{"id":"4569dcca.c9c384","type":"rpi-dht22","z":"a56382b8.978a5","name":"","topic":"rpi-dht22","dht":"11","pintype":"3","pin":"7","x":686,"y":125,"wires":[["7e23f0bc.8c47"]]},{"id":"7e23f0bc.8c47","type":"change","z":"a56382b8.978a5","name":"","rules":[{"t":"set","p":"temp","pt":"msg","to":"payload","tot":"msg"},{"t":"set","p":"payload","pt":"msg","to":"{}","tot":"json"},{"t":"set","p":"topic","pt":"msg","to":"telemetry","tot":"str"},{"t":"set","p":"payload.temp","pt":"msg","to":"temp","tot":"msg"},{"t":"set","p":"payload.humidity","pt":"msg","to":"humidity","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":886,"y":125,"wires":[["a3db9f31.7f825","9fbeb488.eeb368"]]},{"id":"a3db9f31.7f825","type":"debug","z":"a56382b8.978a5","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1306,"y":285,"wires":[]},{"id":"9fbeb488.eeb368","type":"azureiotdevice","z":"a56382b8.978a5","deviceid":"","connectiontype":"","authenticationmethod":"","iothub":"","isIotcentral":false,"scopeid":"","enrollmenttype":"","saskey":"","certname":"","keyname":"","protocol":"","methods":[],"information":[],"gatewayHostname":"","caname":"","x":1120,"y":120,"wires":[["a3db9f31.7f825"]]}]

An instance of Microsoft Azure IoT Central was created and a device model that sends temperature and humidity information was configured.

From the model, an instance of the device was created and it was enough to access the connect link to obtain the necessary information so that Node-RED could connect to the service.

The device ID, the DPS (Device Provisioning Service) scope ID, and the SAS authentication key have been copied and entered into the node configuration.

And this is the complete stream already running, sending real temperature and humidity data to Azure IoT Central.

And here is the data being received, stored and displayed within Azure IoT Central.

Conclusion

This article describes the process of creating a Cluster of Raspberry Pi's using Kuberenetes to create a virtual environment for teaching concepts about the Internet of Things. I hope it inspires other professionals to seek solutions to spread the teaching of these technologies.