UX Prototyping Platform for IoT/Hardware Projects

Description

Even though the Internet of Things (IoT) is a hot topic, with today's tools it is really hard to prototype and test IoT products or services beforehand. Very few tools allow us to use hardware components (like buttons or knobs) together with software user interfaces (UIs). Most software prototyping tools are closed systems offering no or only very little outside connectivity. Most of them only allow us to link screens together and that’s about it.

Plus, hardware prototyping is more complex than software prototyping: In addition to software interfaces, you need to know something about hardware, of course. Programming skills are needed if you want to tinker with them. And some knowledge about IoT protocols and platforms are helpful, too. These are skills very few user experience (UX) designers possess, since a lot of them focus on the design of digital-only products and services. Just google the evergreen argument whether UX designers should be able to code.

Motivation

I thought that this was unfortunate, since prototyping is such a crucial activity for creating successful products or services. We need to make your ideas tangible, test them with users and evaluate their feasibility before starting to build the real thing. And since I have some experience with software prototyping tools and tinkering with hardware I started looking for a way to create IoT prototypes.

The solution I created enables you to rapidly combine hardware and software components using a standard user interface prototyping tool and use off-the-shelf hardware.

Hardware components can send and receive data to a software prototyping tool with it, and vice versa. You can read temperature data from a sensor, or track a hardware button click. Or you could send text to small LCD screen. And the UI prototyping tool allows you to prototype and build user interfaces without programming skills. But the solution could also be used for purposes other than prototyping, for example as a platform for escape room puzzles or to build your own custom IoT dashboard.

What to Expect

Quick disclaimer: all of the components I am going to use are open source. All ideas that make up the solutions can be found online, when you know what you are looking for. Others before me have explored and followed similar paths. And since I am standing on the shoulders of giants, I want to return the favor and share what I learned and explain what I came up with. None of this is private or confidential. It’s a new application of existing knowledge.

Since the solution is made of several parts, I will not describe all the steps needed to build the platform in full detail. This would be too big of a task. Instead, I will explain the underlying concepts and how they are related and will point to more detailed information and tutorials.

Prerequisites

Below are things that will make it easier for you follow along. But none of it is really a must. As stated before you will be able to find additional information online if you look for them.

• Knowledge in UX/UI prototyping
• Understand what a web server does
• Some basic HTML/JavaScript knowledge helps
• Some basic knowledge about electronics and hardware components
• Not being afraid of the command line
• Being OK with researching topics online

With this out of the way, let’s get started,,,

Vision & Guiding Principles

As we learned in UX 101, all products should start with an idea, a vision or statement that captured its essence. Here was mine of building a UX prototyping platform: Be able to build physical(IoT) prototypes, combining hardware and software user interface elements, fast and easy.

Even though this is quite a broad statement, ease of use and being fast were key aspects of the concept. To better capture the concept I wrote down some guiding principles. These were meant to help me make design decisions when building the solution.

• Use an existing software prototyping tool. I didn’t want to overcomplicate existing tools or break the tools workflow when adding hardware.
• Use off-the-shelf hardware to be able to use open source libraries and drivers, and harvest the knowledge of the internet.
• Limit the amount of programming. Make it easy to make changes by non-programmers (read UXers) and reduce the complexity as much as possible.
• Make it self-reliant and robust. The platform should work out of the box. It shouldn‘t rely on an internet connection. You will never know where your prototype will be used. Plus, it should run without any hiccups for an extended period of time.
• Make it modular and flexible. Make it easy to swap or exchange parts of the solution. Make it easy to add parts.
• Allow for a division of labour. Try to separate the software and the hardware prototyping part as much as possible, since often different people with different backgrounds will work on this (e.g. UX designer, creative technologists and developers).

All of these statements explain what makes the solution easy to work with and allows for fast results.

Requirements

When thinking about prototypes that integrate hardware and software prototypes I came up the following needs:

• We want a stand-alone box to be our server. It should boot up and be ready. And it should be accessible remotely since we don’t know how and where we will use it.
• We want a prototyping tool that produces HTML and the HTML needs to be accessible to us. HTML we can extend and tinker with. We can add and enhance it. Tools like InVision are of no use.
• The HTML code must we stored on a web server then we can update and change it on the fly. Plus, we can view it on different devices.
• We need a way to update the HTML prototype based on the data we receive or send from the hardware.
• Thus, we need protocols to transfer and store information between the the hardware components and the HTML prototype.
• We want wireless connectivity between the hardware components and the server. This removes the need to solder or connect wires. The components can then also be located further away from the server box.
• The behavior of the hardware components must be easy to update and to change. So we can still make changes to the prototype without taking everything apart.

These requirements did not only help me make design decisions but also will provide some structure for the notes to come...

(End of part I)

The Platform’s Architecture

Based on the before stated principles and requirements for the prototyping platform I came up with the following architecture:

On the left you see the web server where the HTML of the Prototyping tool will be stored.

On the right, the different hardware components (sensors and actuators, e.g. buttons, rotary encoders, LEDs) are connected to ESP8266 microcontrollers. In broad terms, ESPs are similar to Arduinos but also offer WiFi connectivity for a low price. In between sits a server to broker and translate the information coming from both sides.

The ESPs will send MQTT messages when the user interacts with the hardware components, and receives them when the user interacts with the software part. The (static) HTML page will use the WebSockets protocol to send and receive messages.

These are the key components of the Prototyping platform. In the following, I will discuss individual components and provide additional details.

Source code and how-to will follow.

The Server of Servers and Wireless Connectivity

The two servers shown in the diagram above (the web server and the communication server) are running on a single computer (yes, a Raspberry Pi, as you might have guessed).

The Raspberry Pi is a cheap single board microcomputer. I am still using a Raspberry Pi 3, even though the version 4 is already available. The v3 is fast enough for his purpose.

For the operating system I chose the default Raspberry Pi OS desktop version. (I’ll explain why I chose the desktop version later on, so bear with me.)

Since I wanted to have wireless connectivity to exchange messages and access the web server, I added an additional WiFi dongle to the Pi and configured it to provide an access point. Thus creating its own WiFi network.

Now the ESPs knows which WiFi to connect to and the IP address of the server is known as well. No need to open the web server or other parts of the solution up to the Internet. The core part of the solution needs no Internet connection and sets up it's own WiFi network.

But in case the Pi has an internet connection, I forward the ethernet and the built-in WiFi connection to the access point connections to allow them access as well. This allows me for example to update the devices and access online services.

Note: the easiest way to access and setup the Raspberry Pi is to connect it via an ethernet cable to your computer and share your computer’s internet connection (Mac/Windows). A WiFi connection will drop when you follow the access point setup instructions.

Since the operating system and the data of the Pi are stored on a Micro SD card, it‘s really easy to make backups and to deploy last minute changes. And when you remove the card, all the intelligence of the hardware platform and the prototype are gone.

As you can see, the Pi has several jobs: it is the WiFi access point, it needs to provide a web server, a MQTT broker (which is needed to send/receive MQTT messages), and provides the communication server. An additional admin interface for the communication server would also be nice.

The Pi should share the web server’s folder on the network to allow for an easy way to upload the HTML prototypes. It should also be accessible remotely, to edit and to configure it, if needed.

The Communication Server

The main job of the server is to connect the prototype’s HTML and the ESP devices via MQTT and WebSockets. In a first iteration I used Node.js​ and Express to write it, resulting in about 600 lines of code. But since I am a UX designer and not a developer it's probably best when I don't write (a lot of) code.
The source code felt a bit crude and overly complex to me. It was hard for someone with little coding experience to get how the server worked. And that made maintaining and changing it hard for regular folks.

Luckily, I stumbled over an article describing Node-RED, a visual node-based programming language „for wiring together hardware devices, APIs and online services in new and interesting ways“. And it came pre-installed on the Raspberry Pi. So I gave it a try.

Node-RED FTW

It actually took me a while to wrap my head around the paradigm of node-based programming. But after that, I realized that Node-RED was a brilliant choice.

And after using Node-Red for about three years, I am still in my honey moon with it. Node-RED is powerful, simple, often beautiful, and it’s level of abstraction is just right.

To show you what I mean, here is the "source-code" of the server:

To give you a bit more context of what you are looking at: the purple nodes are MQTT nodes, the olive ones are for WebSockets, the the khaki one on the left is a node which evaluates the messages, and the one on the right stores a value in a variable.

The left flow listens for MQTT messages, determines its content and sends some to the HTML page via WebSockets. The right one does the reverse: it waits for a web socket message, stores it, and sends it out via MQTT.
As you can see, the server consists of six core nodes (that is counting all but the dark green ones, as they are output nodes I use for debugging). And that’s all there is to it.

Detour: MQTT-Messages

Now, a quick detour to MQTT to help you to understand better how the server works…
MQTT is a topic-based protocol. You can subscribe to topics and you can send messages about them. All messages are handled by the MQTT broker. If you send a message about a topic, the broker forwards it to all interested parties (the subscribers). Topics have a hierarchy, using slashes. As an example, a topic for the IoT platform would be: iot/wemos01/button.
All topics of the IoT platform start with the iot prefix, followed by the element ID, and then followed by the (hardware) component. This is a topic the wemos01 would send, when the button connected to the ESP is pressed.
If you now look to the (purple) MQTT node on the left, you see that it listens to the topic: iot/#. The # symbol is a wildcard, thus the MQTT nodes has subscribed to all topics starting with iot/. It receives all IoT platform topics.

With Node-Red, I went from 600 lines of code to only six nodes, which is quite a ratio! The flow is simple and easy to understand. You can look at it and can see what it does. Brilliant, isn't it?

The Web Server

The one piece that is still missing, when we look at the architecture diagram, is the web server. But Node-RED comes for the rescue.
You design your Node-RED flows in your browser. This means, Node-RED has a built-in web server. In its settings you can define a folder for „static“ web pages. This is where we will place the prototype’s HTML and Node-RED will host them for us. Thus, Node-RED covers both the communication and the web server elegantly.

Outlook

This concludes the third installment of the series. I hope you find the pace and the level of abstraction ok.
Stay tuned to see what’s next – as I actually haven’t decided what the next part is about. It’s either about Axure or about the hardware components and the ESPs.
We will see…