ThumbCam - ultra low power tinyML touch controller

See what we touch

Imagine your fingers were encrusted with camera's so small you didn't even notice they were there. Your fingers would literally do the talking as they transparently communicated back to your Augmented Reality (AR) or Human Device Interface (HID) management system in a way that made your whole world come to life at your touch.

All Thumbs

While the fully transparent wearable Ferrari version is somewhat in the future the clunky model-T level version can let us try out at least some of the future possibilities today!

Introducing the ThumbCam ( Concept )

ThumbCam is an ultra-low power 'always on' class camera powered by ultra-low nano-power level microcontroller you wear on your thumbs like a guitar pick. It runs on a cell battery and wirelessly transmits tinyML events.

What's it for?

There are a vast number of buttons, switches, dials, knobs and other mechanical human interface devices.

Wouldn't it be great if you could simply print a picture of the control interface, slip on your thumbCAM(s) and start pressing buttons to test things out without having to physically build it first.

But how do you know what buttons are what?

Well the big idea is you design your HID controls in Virtual Breadboard (VBB) using the VBB component development kit (CDK) and you then just print out your control panel and VBB figures out everything else. Details such as..

• Computing the tinyML configuration from the VBB design
• Connect the thumbCAM's and upload the configuration
• Communicate at runtime to receive tinyML events
• Forward events to virtual or retransmit to real hardware

.. well at least that's the idea..

In case you are wondering, VBB is a prototyping tool for embedded developers which I happen to be the developer of. I have been migrating VBB from being a purely virtual sandbox playground to be becoming a mixed-reality platform and tinyML looks like a great addition to integrate.

In the mixed-reality concept of VBB some things are real and some things are virtual. For example the Avatar family of interface connects popular physical microcontrollers to Virtual Components in VBB so users can mix real and virtual components together while developing their projects.

To illustrate, in the example below a real Arduino ProMicro is rendering to a virtual OLED and is being controlled by a virtual control pad.

Virtual control pad

It's the control pad I want to focus on. What if we could print it out and hold in our actual hand to get a proper 'feel' for the game. I know it's only got 2 buttons but it should make clear at least in principle what the plan is.

When designing a custom component in VBB, tinyML is a control in the CDK toolbox which you can place in a design and then configure to link events to the various visual elements.

In this proof-of-concept only a straight-forward color tracking algorithm is used but as additional AI features are supported they would be configured by using the properties of the tinyML widget..

The goal is to be able to press on the printed buttons using the thumbCAMs and have them perform the same actual function as the real buttons would.

Proof of Concept Color Tracking

Like most things worthwhile there is an entry level ( think dial-up ) all the way to super high end ( think fibre optic). Color tracking can be considered the dial-up modem version of tinyML. It's not broadband but don't forget that it was dialup got the internet party started.

avrCAM

In 2004 an extraordinary real-time processing technique was developed by John Royce Orlando for tracking 8 colors at 30 fps using an 8-bit AVR mega8 from an OV6620 camera chip at 176x144 pixels.

Keep in mind these devices are 1K SRAM and 8Kbytes flash 16Mhz devices. Quite amazing.

You can read more about the avrCAM here

NXTCAM

A version of the avrCAM for The Lego Mindstorms NXT robot was developed by mindsensors called the NXTCam. This was the camera I used for this proof-of-concept project since it was one I found lying at the bottom of my box of bits.

Actually, I found it was so old that I couldn't find the drivers so I had to hack on a serial adapter bypassing the ftdi device to get it to work.

The idea with Color tracking is you sample the colors you want to track and then the avrCAM builds bounding boxes against these saved color profiles on the fly.

The important things for this application is real-time performance because we are trying to support full speed human interaction. Under the right circumstances, with correct lighting and color separation, it does a remarkable job with that task.

Button Press Feature Detection

To actually detect button presses the total volume of area is analyzed. The mechanics are deliberately configured to be low noise and low ambiguity. This is a signal that is quite clear and easily processed on-chip.

Button Event Integration

The avrCAM generates bounding boxes from color data packaged in a straight-foward serial packet protocol. This data is received and decoded by tinyML runtime widget in VBB which routes the events to the correct components using the integrated expression language of the Component Development Kit.

The end result is another example of mixed-reality. The use of machine vision to control a virtual printout of a control panel which then controls it's virtual avatar inside VBB.

From proof-of-concept of prototype

The next step will be to move to a prototype of the thumbCAM.

The initial prototype is planned to be

• ATmega328p (Arduino compatible device)
• Cell battery
• 3D printed thumb pick style plastics
• wireless infrared remote control protocol button event generator
• White LED light to control lighting
• ultra low power camera eg HM01B0
• Kickstarter?

Porting of the avrCAM

To support the extra logic of the signal analysis and and wireless communications protocols the mega8 is too limited. Hence I have ported the avrCAM (attached in resources ) to the ATmega328p and set it up as an ATMEL Studio project. avrCAM is not really compatible with Arduino itself as it reserves the use of Timers in very specific and time-sensitive ways so any libraries used from Arduino will have to be ported over to live within the avrCAM constraints.

Power Consumption

Running from a cell battery with a real-time requirement to maximize fps makes color tracking a good pragmatic choice but somewhat limiting. The main advantage of using the real-time 8-bit processor's such as the ATMega328p is that there are very low power devices in this class. To move to more advance tinyML solutions will require careful balancing of power vs performance.

The main issue of binding in the Advance Touch features is achieving the real-time throughput at low enough power that the call battery is achievable.

Advance Touch Future

Putting the camera at the fingertip brings the opportunity for fine-grained capture of exactly what the finger is up to. It is one thing to have say a half dozen or so distinctly colored buttons, which by the way will work great for many hobby and school projects, but were talking next level here.

Imagine hundreds of buttons in an industry control panel. Then imagine that the industrial designer decided that they didn't need any real physical buttons at all! That instead they would go all printed control panels throwing away mountains of wires and mechanical buttons.

No pressure, but now the fingertip camera's better work. So where is the limit?

Well as the finger approaches a specific button maybe there is text on that button. Maybe the text is a bit grubby from being touched and not cleaned in a while.

The advantage of the thumbCAM it is has direct visual access to the button. Any AR system would only be able to infer and may have it view incumbered. But in combination with AR leads to very power possibilities. Anyhow the thumbCAM can actually read the text or the QCODE or the binary color codes on the actual button. However the alignment won't be 'machine repeatable grade. It won't be lined up like in some machine setting where text it directly at known and unchanging positions.

So this sort of noisy orientation with variable image quality is going to require some serious pre-compute and cognition on demand pushing tinyML to it's limits. , Can 30fps with a budget of only a few milliamps be achieved?

My guess it will be achieved step by step. Starting from higher constraint but limited solutions like color tracking on 8-bit micros and growing from there.

Supporting Testing Support

To test the ATmega328p port I built a virtual OV6620 device that I can use from the Avatar Shield. A side benefit is this can be used together with an Arduino to build didactical experiences for developing real-time processing on bare-metal.

In the future I will modify this to support other camera types (low pixel count) and also enable it to render visualizations of the what the real camera will see in active use.

This can be used to generate training scenarios for tinyML AI networks

Thanks!