Published June 6, 2022 © CC BY-NC-SA

Nikola: Privacy-Focused Camera Bot

A cute little companion bot for media excursions! With a smart, consent-based camera, high-quality microphone, and cute wiggly ears.

IntermediateWork in progress10 hours2,650

Things used in this project

Hardware components

Arduino Nicla Vision

Adafruit Gemma M0

LED (generic)

SG90 Micro-servo motor

USB-A to Micro-USB Cable

Software apps and online services

OpenMV IDE

Hand tools and fabrication machines

Soldering iron (generic)

Story

A privacy-focused event companion for Embedded Vision Summit 2022! This little bot will take your photo, but only when the QR code for this project is in the frame. :)

One of my first images taken with the consent QR code!

This bot runs on the Arduino Nicla Vision, with an Adafruit Gemma M0 running the ear motors. The whole thing started when I realized that my Companion Core PCB fit perfectly inside a crappy phone-holder I had, and would work well with a little tripod – AND I could stick it in my shirt pocket! THEN I realized that both pieces fit well with the head assembly I'd been designing for another robot, and this prototype just kinda came together.

It also happened to be the fourth anniversary of finishing my first robot familiar, Archimedes, on the eve of Bay Area Maker Faire 2018! So, this felt pretty magical. :)

Vision system

The Nicla is set up to read any barcodes in the frame, using the OpenMV IDE and compatible MicroPython code, following this code from Adrian Rosebrock: https://pyimagesearch.com/2018/03/19/reading-barcodes-with-python-and-openmv – You can find my mashed-up version in the Code section below.

Once it finds a barcode, if it matches this URL (https://bit.ly/furrybot), it illuminates the red LED (both the onboard LED and one I've attached to GPIO), takes a photo, and saves it to local memory with an incrementing counter. On this prototype, it currently resets the counter each time it wakes up, so I'll need to rewrite that to prevent new photos from overwriting the older ones. As it stands, the photos become readable on the board after it resets, which is handy because I can pull them onto my iPad using it as a USB flash storage device. I used this to back up the photos during Embedded Vision Summit.

The Nicla is hooked up to a right-angle Micro USB cable, which I plug into a battery pack stashed in my bag.

Ear servo system

The servos are mounted in a "sandwich" between two layers, which are currently 3D-printed but could be laser-cut in the future. They're mounted at 45º angles, which helps with making lifelike animations (as seen in my fennec fox robot, F3NR1R).

This bot started out as the first one to use my Companion Core! However, I couldn't get both of the ear-servos to work at the same time, which I now believe is due to something in the code or power system of the Gemma M0.

Still, while ruling out hardware issues, I decided to cobble together a little protoboard so I could make sure it wasn't a problem with the Core. This board connects the Gemma (powered by its own LiPo) to the servos. The whole thing is hot-glued to the back of the phone mount.

I wrote the code in MakeCode block script, which you can grab here: https://makecode.com/_KRJWw9Hk97tE

Although both servos have worked at different times, I couldn't get it to be reliable, so for this conference, I dialed it back to flicking just the right-side ear. Room for improvement!!

Microphone

This little 'bot is ALSO inspired by those "dead cat" wind-screening fuzzy sock things they put over microphones, and serves much the same purpose. It holds a cardioid lavalier mic, and can be held like a regular microphone when I want to do interviews. (Unfortunately, if I haven't secured the cables, those can cause noise in the recording... as I found while trying to edit some of my footage! Alas! In future, I'll wind the cables around my arm and try to minimize moving the mic.)

I designed this mounting block in OnShape (project link), and while I had some slicer/printer issues at first, I got a really solid one eventually. Here's a fit test on a failed, crumbly one, and you can see the final version mounted below the Nicla in the photo under "Vision System":

It slips between the bars of the phone holder, and straddles the outside of the servo sandwich.

Next steps

Next, I've gotta fix that photo file-naming issue! And figure out how to get both ears working... I'd also like to get them running on the Nicla Vision itself, but the documentation for that is scant.

I also need to figure out framing on the photos. Unfortunately, there's no image preview (unless you have it hooked up to a laptop), and so it's really hard to get A) the code and B) a person in-frame! There's a lot of dancing around and hoping, and the results are... interesting. A bit like a wildlife camera trap 😅

Here are some shots from the conference! Judge for yourself!

1 / 5 • Yours truly, barely in-frame and definitely moving too fast

Short link to this page: https://bit.ly/furrybot

Code

# Adapted from: https://pyimagesearch.com/2018/03/19/reading-barcodes-with-python-and-openmv
# And: https://docs.arduino.cc/tutorials/nicla-vision/getting-started#using-the-nicla-vision-camera
# by Alex Glow for Hackster.io, 2022
# for F3NR1R: http://bit.ly/fenrir2

# import necessary packages
import sensor
import time
import image
import pyb

sensor.reset() # Initialize the camera sensor.
sensor.set_pixformat(sensor.RGB565) # Sets the sensor to RGB
sensor.set_framesize(sensor.QVGA) # Sets the resolution to 320x240 px
sensor.set_vflip(True) # Flips the image vertically
sensor.set_hmirror(True) # Mirrors the image horizontally

# initialize the clock
clock = time.clock()

redLED = pyb.LED(1) # built-in red LED
blueLED = pyb.LED(3) # built-in blue LED

ledPin = pyb.Pin(pyb.Pin.board.PG1, pyb.Pin.OUT_PP)
ledPin.on()
pyb.delay(500)
ledPin.off()

redLED.on()
sensor.skip_frames(time = 2000) # Skip some frames to let the image stabilize  about 4 sec
redLED.off()
blueLED.on()

snapno = 0

# barcode type lookup table
barcode_type = {
    image.EAN2: "EAN2",
    image.EAN5: "EAN5",
    image.EAN8: "EAN8",
    image.UPCE: "UPCE",
    image.ISBN10: "ISBN10",
    image.EAN13: "EAN13",
    image.ISBN13: "ISBN13",
    image.I25: "I25",
    image.DATABAR: "DATABAR",
    image.DATABAR_EXP: "DATABAR_EXP",
    image.CODABAR: "CODABAR",
    image.CODE39: "CODE39",
    image.PDF417: "PDF417",
    image.CODE93: "CODE93",
    image.CODE128: "CODE128"
}

def barcode_name(code):
    # if the code type is in the dictionary, return the value string
    if code.type() in barcode_type.keys():
        return barcode_type[code.type()]
    # otherwise return a "not defined" string
    return "NOT DEFINED"

# def ninety()
#    pyb.udelay(1500)

# loop over frames and detect + decode barcodes
while True:

    # tick the clock for our FPS counter
    clock.tick()

    # grab a frame
    img = sensor.snapshot()

    # loop over standard barcodes that are detected in the image
    for code in img.find_barcodes():

        # draw a rectangle around the barcode
        img.draw_rectangle(code.rect(), color=127)

        # print information in the IDE terminal
        print("type: {}, quality: {}, payload: {}".format(
            barcode_name(code),
            code.quality(),
            code.payload()))

        # draw the barcode string on the screen similar to cv2.putText
        img.draw_string(10, 10, code.payload(), color=127)

    # loop over QR codes that are detected in the image
    for code in img.find_qrcodes():
        # draw a rectangle around the barcode
        img.draw_rectangle(code.rect(), color=127)
        # print information in the IDE terminal
        print("type: QR, payload: {}".format(code.payload()))
        # draw the barcode string on the screen similar to cv2.putText
        img.draw_string(10, 10, code.payload(), color=127)
        # save pic & delay 2s if our QR code is detected

        if (str(code.payload()) == "https://bit.ly/furrybot"):
            blueLED.off()
            redLED.on()
            ledPin.on()
            print("code found! saving photo: ")
            imgname = str(snapno) + "-qr.jpg"
            sensor.snapshot().save(imgname)
            snapno = snapno + 1
            sensor.skip_frames(time = 2000) # Skip some frames to debounce
            redLED.off()
            ledPin.off()
            blueLED.on()