People interact with a handful of physical buttons and touch screens every day (when they're actually out and about in the world :)). Given the new normal with COVID-19, these user experiences just aren't practical anymore. To solve this problem I want to create a device platform that can be used in public spaces while still being controlled by users and their smart phones. This project aims to address the following issues:
- Device platform should be highly customizable to support a variety of equipment (i.e. the stamp machine at the post office, washing machines, automated car washes, soda dispensers in a food court). Customization means the device should be simple and able to hook up to already existing equipment rather than being a full replacement. This is mostly solved by using the Arduino platform.
- Users should be able to connect to the device for a short period. The device should use a method to determine if the user is 'close enough' to start using the device (such as RSSI distance measuring). Device should routinely evaluate if a user should still be connected, or if they should be automatically disconnected (either after the user moves away from the device, an action is fulfilled, or a set amount of time of inactivity has passed).
- User should have a single control app with customizable controls determined by the device they want to interact with. Device should inform the controller app which controls (with expected ranges) are needed to operate it.
I will use a single, simple, example to develop this platform and demonstrate the idea using an app and physical device that can turn an LED on and off - anything more is just details. The end product isn't the important part of this idea, but the core user experience is.
Project Technical SummaryThis project works by using a simple hardware board (ESP32 for easy-to-use BLE) that can be hooked into older, non-connected, devices. That device can be connected to with a mobile app using Bluetooth. Once the devices are communicating, an ID can be requested by the mobile device that pulls down a JSON string from Firebase Real-Time Database so that a UI can be dynamically built on the mobile device for controlling that specific device. The mobile app is written using Flutter so it will work on both iOS and Android.
The bluetooth connection code partially came from this project that I used to figure out how everything should communicate back and forth.
How Would This Work as a Business?Rather than being a purely B2C focused business, I see this as more of a service. Using the same simple hardware and universal app we could provide a device for retrofitting existing machines to handle touch-less interaction, as well as trainings for a company's technicians to install the device in their pre-existing hardware.
Hardware SetupThe hardware setup is incredibly simple: I used a single ESP32 to interact with the mobile app, and any peripherals (original 'dumb' machines) would be controlled by that ESP32.
Arduino CodeThe code for this project on the hardware side is deceptively simple. I'm using the Arduino IDE and some BLE libraries specifically for the ESP32 to broadcast as a BLE device, and then send information back and forth between the mobile app and the device. When a message is received, it gets checked to see if it's a command that is expected in the WriteString
method. If it's 'request_id
', then the ID for a Firebase Real-Time Database node is sent back to the mobile app to download a specific JSON string representing the UI that would be used to control this device. If it's an action that can be taken by the device, then that action is taken.
#include <BLEDevice.h>
#include <BLEUtils.h>
#include <BLEServer.h>
#include <BLE2902.h>
#define DEVICENAME "PROTO"
#define SEND "9c81420d-8a1e-49f8-a42f-d4679c7330be"
#define SEND_STRING "1394b0cc-e9b7-4cc1-bc50-d6c9a9505f21"
#define RECIVE "f159cdfb-df60-4a4a-b1fa-004bcc379bb6"
#define RECIVE_STRING "cd468881-1cda-47a1-9373-dc812d15d727"
#define UI_ID = "dfa653";
bool deviceConnected = false;
BLECharacteristic *sSendString;
String strToString(std::string str) {
return str.c_str();
}
int strToInt(std::string str) {
const char* encoded = str.c_str();
return 256 * int(encoded[1]) + int(encoded[0]);
}
double intToDouble(int value, double max) {
return (1.0 * value) / max;
}
bool intToBool(int value) {
if (value == 0) {
return false;
}
return true;
}
class ConnectionServerCallbacks: public BLEServerCallbacks {
void onConnect(BLEServer* pServer) {
deviceConnected = true;
Serial.println("Connected");
};
void onDisconnect(BLEServer* pServer) {
deviceConnected = false;
Serial.println("Disconnected");
}
};
class WriteString: public BLECharacteristicCallbacks {
void onWrite(BLECharacteristic *pCharacteristic) {
String str = strToString(pCharacteristic->getValue());
sSendString->setValue("default");
Serial.print("Recived String:");
Serial.println(str);
if( str == "request_id") {
Serial.println("sending response: request_id!");
//randomly generated device UI identifier
sSendString->setValue(UI_ID);
sSendString->notify();
} else {
//handle action! - this is where we'd interact with peripherals
Serial.print("we got a new action: ");
Serial.println(str);
}
}
};
void setup() {
Serial.begin(115200);
Serial.print("Device Name:");
Serial.println(DEVICENAME);
BLEDevice::init(DEVICENAME);
BLEServer *btServer = BLEDevice::createServer();
btServer->setCallbacks(new ConnectionServerCallbacks());
BLEService *sRecive = btServer->createService(RECIVE);
uint32_t cwrite = BLECharacteristic::PROPERTY_READ | BLECharacteristic::PROPERTY_WRITE;
BLECharacteristic *sReciveString = sRecive->createCharacteristic(RECIVE_STRING, cwrite);
sReciveString->setCallbacks(new WriteString());
BLEService *sSend = btServer->createService(SEND);
uint32_t cnotify = BLECharacteristic::PROPERTY_READ | BLECharacteristic::PROPERTY_WRITE |
BLECharacteristic::PROPERTY_NOTIFY | BLECharacteristic::PROPERTY_INDICATE;
sSendString = sSend->createCharacteristic(SEND_STRING, cnotify);
sSendString->addDescriptor(new BLE2902());
sRecive->start();
sSend->start();
BLEAdvertising *pAdvertising = btServer->getAdvertising();
pAdvertising->start();
}
void loop() {}
Currently the app has a button for connecting to the hardware device, and a dialog where the device name can be changed for connecting to different devices. The next step will be adding a QR code reader for identifying specific devices for controlling. All of the code files are attached to this project page, but the important block is here:
List<dynamic> view = null;
Widget getViewForItem(String data) {
if( view != null ) {
//TODO: magic of casting to object types is here. Add style attributes and other properties to make this prettier later.
List<Widget> widgetList = List();
for( final item in view ) {
if( item != null ) {
print("Item to parse: " + item.toString());
switch(item["view_type"].toString() ) {
case "text": {
widgetList.add(Text(item["text"].toString() ));
break;
}
case "button": {
widgetList.add(RaisedButton(
child: Text(item["text"].toString()),
onPressed: () {
bluetooth.sendTextFieldValue(item["action"].toString());
},
),);
}
}
}
}
return Expanded(
child: SizedBox(
height: 200.0,
child: ListView(children: widgetList) )
);
} else if( data == "scanQRCode" ) {
return Text("QR code reader goes here");
} else {
FirebaseDatabase.instance.reference().child('request_id').child(data).once().then((DataSnapshot snapshot) {
view = snapshot.value;
(context as Element).reassemble();
});
return Text("Loading...");
}
}
This code segment receives a UUID for a UI block that's stored in Firebase (the request for it is sent as soon as the mobile app has connected to the hardware device) and then dynamically builds the UI based on that block. I've kept it super simple for now (just a single text view and a single button), but it should be highly customizable via this parsing and building system. Once this is fleshed out, a person with this app should be able to control any similar device with a UI that matches up to the necessary functionality.
It's worth noting that the button's action
value, toggle
, can be seen on the serial monitor screenshot above for the ESP32.
The code behind reading QR codes with Firebase Machine Learning is a bit much for this post. For a really good example, here's the FlutterFire library, which is what I used for my prototype.
While the device and app work well right now as a prototype/idea, there's definitely a few things to add over time.
- Measuring RSSI to determine if someone is close enough to the hardware device to be able to connect to it.
- Idle timer for auto-disconnecting a user.
- QR codes would need to be printed specifically for each device. Each hardware device needs a bit of a personal touch
- Cutting out FIrebase Real-Time Database would be preferred for this use-case, but I was having trouble sending larger chunks of data over bluetooth, so I stuck with sending a UID over a characteristic and using a tool that I knew would work well and easily :)
- Analytics on the hardware device would be great. Could be used for preventative maintenance, and keeping track of supplies for things like soda machines.
Comments
Please log in or sign up to comment.