Abhishek Singh’s Augmented Reality App Provides a Fascinating Way to Interact with IoT

The popularity of IoT devices and smart homes is rapidly increasing, and yet we still haven’t come to a widely-accepted decision about how best to interact with them. For the time being, most of us are still stuck shouting at Alexa with the hope that she’ll agree to turn off a light for us. In the cyberpunk future many of us long for, we’ll all have augmented reality eye implants so we can simply look at that light to turn it off. Abhishek Singh’s DIY Raspberry Pi-based smart home system is as close to that as we’ve seen, and works today.

Singh’s project has garnered a lot of attention over the past couple of days in the form of a viral tweet, and we got a chance to speak with him about the details. That tweet, shown above, includes a video of Singh demonstrating his build’s functionality. Using an iPhone, he can look around his room through an augmented reality app he wrote. Whenever a smart device comes into view, “on” and “off” buttons pop up on the screen next to the device, and pushing those buttons toggles power to the real, physical device. As you’d expect, there is a lot of technology whirring in the background there, so let’s dig into how it works.

Most of the magic here is happening within the iPhone app that Singh programmed. It was built in Unity3D, and utilizes Apple’s ARKit functionality. Singh compiled a small library that includes images of the devices he wanted to control. The app uses image detection to identify those devices, which is much quicker than ARKit’s object detection. Once a device has been identified, the augmented reality buttons are overlaid on the screen.

When one of those buttons is pressed, the app makes a REST API call to a Node.js server running on a Raspberry Pi Zero W. The Raspberry Pi, in turn, toggles the corresponding device’s power relay through the GPIO pins. The resulting effect is what you see in the video: virtual buttons shown through augmented reality that control a smart home. It’s not quite cyberpunk eye implants, but it’s as close as possible with today’s technology.