Secure Computer Vision & Voice Connected Fridge

Ever wanted an automated way to manage what you need and what you have when your out at the store?

Lists, notepads, whiteboards - those are for organized people. I am not an organized person.

Enter technology...

Let's use some computer vision, voice control and some free services to give you on demand access to what you got and what you need.

At first glance you may think I am over-complicating things, why not just put a web server somewhere and serve my list from there? Security, yes the four letter word to anyone prototyping in IoT today, the elephant in the room, the thorn in the side to anyone who ever plugged in an ethernet cable or gave some device the SSID and password to their wifi.

I'm a big fan of Costner's Law:

So my theory on security is along the lines of, if it ain't there; it's not very interesting to hack.

You may have heard of the term "at the edge", what this means to me in this design is there isn't some great central repository of data or personal information someone can find exploit. There lots of data but it out there distributed on the edge of many devices, expensive and time consuming to steal. To put it bluntly, if I could hack into one server and get a million IDs, that's a pretty attractive target; if I individually have to hack into a million devices, that's a lot of work. Something to consider, but what I was going for here in my design.

Additionally, the data that is your shopping list is not an endpoint, it is accessible by exchanging data + keys through a service where data is never at rest. There is no "there" to get to, no web server to compromise, no obvious target.

Of course, these theories are my own and you can agree or disagree as you see fit. I'm happy to discuss and hear your opinion as well.

Basic Setup

This project code is hosted at github @ https://github.com/plangdon/smart-fridge

It will evolve and improve with time and please feel free to reach out to me with questions or comments.

The code is written in JavaScript and uses some libraries and services where you will have to get account to use. Free accounts are available and should satisfy individual usage.

Computer Vision

We are utilizing a service by a company called ClarifAI (https://www.clarifai.com/). This is a trainable computer vision/recognition service. They have some predefined object libraries, we will be leveraging the "Food" library. You can also train your own vision set but that is outside of the scope of this example. Camera us connected via USB.

Voice Control

We are utilizing a voice recognition library that runs in JavaScript in a browser. It is called annyang (https://www.talater.com/annyang/). It allows you to use a browser window and a microphone to capture and recognize speech.

Messaging

We are using a cloud mqtt service called CloudMQTT (https://www.cloudmqtt.com/). Free accounts are available with usage limits in the thousands.

Usage:

In the home from a browser (tested with Chrome):

Using voice interface, say "add bananas" to add bananas to your shopping list.

You may substitute the word "bananas" with any item you want to add your shopping list. i.e. - saying add "apples" will add apples to your list.

Using the computer vision interface, holding the object you want to add to your shopping list in view if the camera and say "capture", the camera will capture a photo and analyze and recognize the captured photo. Be aware that object recognition may not be prefect as it is based on training the AI to learn objects.

Using the smart phone interface, browse to your shopping list endpoint, click the button asking for current list. Be aware the secure communications channel relies on messaging and you may experience a few second delay to receive shopping list, my experience is it is usually milliseconds.

LinkIt Smart 7688 Duo

MPU and MCU. Powered by Media Tek MT7688 and ATmega 32U4.

What is LinkIt Smart 7688 Duo (from Seeed)

LinkIt Smart 7688 Duo is an open development board compatible with Arduino Yún sketches, based on the OpenWrt Linux distribution, MT7688 and ATmega32u4. The board is designed especially to enable the prototyping of Rich IoT Application for smart house or office. Since it is compatible with Arduino, this allows you to use different features from Arduino Yún and LinkIt Smart 7688 Duo, which will help you build rich applications based on various, robust and compiled Arduino Yún sketches. The board offers you the memory and packet storage to enable robust video processing. The platform also offers options to create device applications in Python, Node.js and C programming language.

LinkIt Smart 7688 duo is a co-design product by Seeed Studio and MediaTek. It brings together the parties' knowledge in open hardware and industry leading reference designs for IoT devices to create this powerful development board.

LinkIt Smart 7688 duo is fully supported on the MediaTek Labs site: check out the LinkIt section and register here to download the SDK, participate in the Forums, and more.

Features

• 580 MHz MIPS CPU

• Single input single output(1T1R) Wi-Fi 802.11 b/g/n (2.4G)

• Pin-out for GPIO, I2C, SPI, SPIS, UART, PWM and Ethernet Port

• 32MB Flash and 128MB DDR2 RAM

• USB host

• Micro SD slot

• Support for Arduino (ATmega32U4)