SmileMirror

Project Overview

People spend about 5 hours a week looking at themselves in the mirror. This is, for most of us, time we spend with ourselves: recapitulating the events of the day, getting ourselves ready to go out or simply admiring our reflection. But to others, this is not so easy.

Studies show that 2 in 5 people always feel dissatisfied with themselves when they look in the mirror. These people would avoid a mirror at all costs and do not enjoy the “me time” that the rest of us do. The same study also states that out of the 40% of people that dodge their reflection, most of them are women.

I am sure that at some point, we all felt agitated to correspond to other’s expectations and to fit into our social group. That constant looking in the mirror and questioning our appearance is stressful. Sometimes we walk out of our house thinking that we look terrible.

Women at menopause are very susceptible to the feeling of disappointment when looking in the mirror. Studies show that women can gain between 1 to 2 kilograms during perimenopause (though many gain more). This together with a potential increase in emotional instability can cause women to despise themselves, avoiding their reflection at all costs.

What is even more alarming is that 1 in 8 adults has suicidal thoughts inflicted by their body image. Looking in the mirror for these people is dreadful, but what if we convince them that life goes beyond what they look like, that it does not matter how your hair looks, that everyone is beautiful in their own way.

This is SmileMirror. I decided to simplify my approach to this problem and thought about what people would want to hear when they look in the mirror. “You can do it” or “Keep going” perhaps. I thought of it like when you learn on Duolingo, and every time you get 5 or 10 answers right in a row, Duo pops up and felicitates you. Although I am aware that Duo is an animated bird, I do feel proud of myself when I am complimented.

What if we could bring this positive feedback to a mirror? This is what SmileMirror aims to do. The application consists of an AI driven smart camera placed on the mirror providing on the edge facial recognition. When the person spends time in front of the mirror, the device detects this and sends them a notification on their phone with an inspirational message to keep them going through the day.

Although simple, this application can truly boost one’s impression of themselves as well as making them feel appreciated. Let’s make looking in the mirror enjoyably for everyone!

What is SmileMirror?

SmileMirror consists of an AI empowered smart camera mounted on the user’s mirror. This camera has a facial recognition module on edge meaning that no image data is sent outside the camera. The camera is paired with an Arduino microcontroller which checks if the camera detected a face.

If a face is detected for 2 minutes, the device will choose an inspirational message stored locally and send it to the user via a mobile notification powered by the Blynk backend. The device will then wait until the person is not in the mirror anymore and will then start the loop again.

One compliment at a time, SmileMirror can work to the benefit of the user.

Going further, what could it be?

I feel like there are promising developments that could be made to this product to further its success and allow it to be mass produced. Below are a few improvements I thought of when developing the product. These improvements could be implemented in a second version of the project.

• Adding facial expression recognition on the edge to the device allowing it to target its message if the user appears to be down.
• Using industrial components reducing the overall cost of the product (for example, replace the Arduino Yun with a cheaper alternative)

Getting to the details

I like to think of my projects as having a frontend and a backend: the frontend being the IoT device and the backend being the application and processes happening in the cloud. When detailing this application, I split it into these two sections.

Frontend

The frontend of the project consists of the Arduino microcontroller and the Huskylens camera module. The camera module is on all the time, powered by the Yun. The camera is already trained to detect and register faces which is what it does while on. If a known face is detected, the camera module will tell the microcontroller that a known face is identified.

If the microcontroller receives this data for 2 consecutive minutes, it will realise that there is a person looking in the mirror (this is done to prevent sending a message when a person is passing by the mirror). The microcontroller will then send a packet of data to Blynk with the inspirational message randomly generated on the device.

I chose to use the Arduino Yun because it is very easy to use. This allowed me to develop an application in a few hours thanks to its onboard WiFi and connectivity options. Huskylens is a smart camera system developed by DF Robot. The device debuted on Kickstarted earlier this year (or last year, I can’t remember). When I heard of this camera, I rushed to get it.

Honestly, it is a nice device, facial recognition works very smoothly (considering the fact that it is done on edge). It can connect to any microcontroller using either serial or I2C. The camera can easily be trained to recognise a face, object, or colour. This is literally done by pressing a button.

Both devices are quite expensive but keep in mind that this is a prototype and I had chosen to use these devices because it allowed me to produce the application faster. There are cheaper alternatives to them available that would be considered if mass producing.

Backend

The backend of the project simply consists of Blynk’s infrastructure which relays the message from the IoT device to the mobile application. The Arduino Yun sends a string containing an inspirational message to Blynk. This message is then relayed onto the application which sends it as a notification to the user. The application can be created with ease and then deployed on iOS and Android.

Market Research

Market research has been briefly conducted to identify competition and establish the product’s USP. Below is a summary of the market research conducted.

Competition and Similar Products

There are several similar products on the market which can be found by searching for “smart mirror” on Google. The problem with these products is that they are very focused on a complex system where data such as temperature and weather forecasts is displayed on the mirror.

As opposed to these products, SmileMirror aims to have a simple functionality which simply sends the user an inspirational message if they look in the mirror. This simplicity is unique among these designs which aim to display a lot of information on the user’s mirror.

Unique Selling Point

The application’s USP is the approach to put humans first and develop a system where people can be uplifted when they look in the mirror, improving their respect for themselves and their self-worth.

User Persona

The persona of the typical person is a woman aged between 40 and 50. Although this device is targeted at women, most people can benefit from it. The user would preferably have a moderate income and be experiencing problems with their body image.

Project Conduct

I have summarized all the steps needed to create this project below.

Let’s get things ready!

First of all, we need to get the components we need for the project. These components are listed out below.

• Arduino Yun Rev2
• DF Robot Huskylens
• Power bank
• Wires
• Breadboard

Alternatively, you could use an Arduino Uno coupled with an ESP8266 module. Please revise the code in this case or send me a message if you need any help doing this.

Setting up the camera

The first thing we need to do is set up the camera module. We want to set it up so that it recognises our face when we look at it.

This link is in the documentation provided by DF Robot and explains how a face can be registered with the device. Please follow the steps provided in the document above and have a face memorised with the camera before moving on to the next step.

The hardware side

Now that we have the camera set up, we can move onto the Arduino Yun. If your Arduino Yun is out of the box, you will need to set it up and connect it to WiFi. This page released by Arduino can guides you through the setup.

After your Arduino Yun is up and running, the next thing to do is wire the circuit. Below is an image with the way the camera should be connected to the Arduino Yun. For this project, we will communicate with the camera using Serial.

Excuse the fact that an Arduino Uno is included in the schematics above, unfortunately, DF Robot did not have a fritzing part available for their Huskylens module and hence I had to use a wiring schematic published on their website.

Let’s look at the code

So now that our circuit is up and running, let’s take a look at the code needed for this project. I compiled a program using the Arduino IDE which is linked below in the code section. In this part, I will walk you through the main functions in the code and I will explain what they do and put them in context.

// CHANGE: Insert your Blynk token here
char auth[] = ""; // insert project token

// define messages
int numberOfMessages = 4; // set manually to reduce use of memory
String inspirationalMessages[] = // you can change the messages just be aware of the memory on the device
{
    "Keep Going!",
    "You can do this!",
    "You’re doing great!",
    "Show them what you've got!"
};

This piece of code is found at the start of the program. The first variable needs to be edited by the user to allow the device to connect to the custom Blynk application. Then the device will store several inspirational messages as an array. You can change these messages but beware of the storage usage on the Arduino Yun as it is quite limited if not expanded.

// globals
float waitTime = 120; // time to wait until sending message (seconds)
bool counting = false; // are we counting time?
time_t currentTime;
time_t startTime;
int errorNr = 0;
int errorAdmital = 25; // quit on 25 (cannot be bigger than waitTime)

More variables, these are very important and although all of them can be changed by the user, it is crucial to understand what each one controls.

• waitTime The amount of time in seconds after which, if a face was present all along, a notification is sent to the user
• counting Stores if the 2 minutes are counting or not. Counting starts when face is seen
• currentTime The current time in UNIX. From Blynk RTC
• startTime The time at which counting started in UNIX.
• errorNr The number of reads where a face was not detected by the camera
• errorAdmital The number of reads without a face detected admitted before the device abandons counting.

// get the data from the camera module
void processCamera()
{
    if(huskylens.request())
    {
        if(huskylens.available())
        {
            while(huskylens.available())
            {
                HUSKYLENSResult result = huskylens.read();
                processResult(result);
            }
        }
        else
        {
            errorNr++;
            if(errorNr > errorAdmital) // if face not detected for a while, stop counting
            {
                startTime = 0;
                counting = false;
            }
            if(errorNr > 1000) // reset value to prevent overspill
            {
                errorNr = 0;
            }
        }
    }
}

This is the loop that is constantly run by the microcontroller. The loop reads the data outputted by the camera module. The device checks if the module is connected and whether it is reporting data. This loop also takes care of error counting; if there is no face detected by the camera, the number of errors will be incremented.

// process the data coming from the Huskylens module
bool processResult(HUSKYLENSResult result) // only runs if block detected
{
    if(result.command == COMMAND_RETURN_BLOCK) // if block returned
    {
        if((currentTime - startTime) > waitTime) // check if the waiting time is over
        {
            if(counting) // if so, send the phone notification
            {
                Serial.println("");
                Serial.println("[loop] Face Identified for " + String(waitTime) + " seconds");
                Blynk.notify(getMessage()); // send the inspirational message
                delay(30 * 60 * 1000); // Set to 30 minutes. If device does not resume running the code, consider using an alarm from time library
                errorNr = 0;
                counting = false;
            }
        }
        if(result.ID == 0) // if face not recognised
        {
            Serial.println("Face Not Recognised ID: " + String(result.ID));
            if(counting)
            {
                errorNr = 0;
            }
        }
        else // if face is recognised
        {
            Serial.println("Face Recognised ID: " + String(result.ID));
            if(!counting) // if the face was just noticed, start counting
            {
                startTime = now();
                Serial.println("face set at" + String(startTime));
            }

            counting = true;
            errorNr = 0;
        }
    }

return true;
}

Now this loop can seem a bit confusing, but its function is simple; check if the 2 minutes are up and a notification is to be sent. As well as that, the loop starts counting the 2 minutes if a face was noticed and resets the number of errors returned.

Compile the code

The next thing to do is open the code in the Arduino, plug the Arduino Yun in and upload the application onto the device. It is important to uncomment the last lines in the void loop() as this will dump all variables to the serial monitor.

When debugging, try setting the waitTime to 5 seconds and putting your face in the frame for that duration of time. Keep the serial monitor open and if you get a print saying that data is being sent to Blynk, you are done!

If you did not succeed, don’t worry. The Huskylens module has an RGB LED on the back that turns green if it detects a memorised face and blue if it detected a face that it is not familiar with. If it is blue, ensure that your face is registered. If it is off, ensure that the camera is in face recognition mode by spinning the wheel (on top of the camera, to your left) until Facial recognition comes up. If nothing works and you simply do not know what to do, feel free to send me a message.

Getting the Backend sorted

Now let’s work on the Blynk side of things. You can get the application for both Androidand iOS. Once you have the application installed, follow the steps below to get going. Create an account and make sure you are signed in before continuing.

Open the Blynk application, you should see a screen such as the one above (probably without a project space).

Next, swipe to the right until this screen shows up, click on the New Project Button to create a new project.

Now give the project a name and select the Arduino Yun from the dropdown menu. Press on Create.

You will now be directed to a blank canvas where you can build your app. Click on the canvas or swipe to the left to show the widgets available.

From the list, select the Notification and RTC widgets one after the other and place them on the canvas.

You should end up with something like this. Now the last thing you need to do is click on the gear icon at the top of the screen and press on the Copy All button to copy the authentication token to your clipboard (an email containing it was also sent to you). Now paste it in the code in the variable auth[] at line 30.

Press the start button at the top right of the screen to start running the application. You are ready to go!

Bring it all together!

Finally, run the debug code once more and you should get a notification on your phone with the inspirational message. Eureka! Now comment out the debug at the end and upload the final code for distribution on the device. All that is left to do is put it on the mirror.

Let’s mount it

Finally, the part that requires little effort (hopefully). I was lucky enough to have a surprising amount of space between the wall and the back of the mirror allowing me to throw all the components in the back leaving only the camera on the front of the mirror.

This may be different for you. If you are really into the project, consider a 3d enclosure for the device. If the device were to be mass produced, there would obviously be a need for consistency across all designs, so a 3d printed enclosure if probably best.

Reflection

Well, this was a fun project (for me). I joined the contest very close to the deadline, I was busy studying but then coronavirus came along and there is nothing to study for anymore. Anyways, I needed to build something out of stuff I had lying around at home. Huskylens was the first thing that came to mind because I did not get a chance to use it before.

So, I thought of an idea for the contest, that would preferably target mental wellbeing and came up with a smart mirror design. I initially wanted to write the inspirational message on the mirror and light it up when someone is looking at it, but because of the time restrictions I had to rethink and came up with the idea of sending the user a notification with the message.

Overall, the project went quite smoothly, I think that it is mass producible and perhaps the whole device can be mounted onto a PCB and shipped that way. But anyways, I decided to target multiple symptoms that were brought on by menopause but not make a product that was exclusive to women in that period. SmileMirror, although targeted at women in menopause, can be used by everyone.

I must admit that the biggest challenge I faced when developing the product was working with variables. I have not really coded in c++ for a while and it was quite difficult getting back into variable types after playing around with JS and python. I spent around an hour looking for ways to store the UNIX time on the Yun (because the Yun stores all numerical variables at a smaller precision than other microcontrollers). But other than that, it was an enjoyable project. I hope you enjoyed it!