Gesture Controlled Smart Mirror

A Smart Mirror is a two-way mirror with an inbuilt display behind the glass. The display can show anything you want on the mirror’s surface such as the current time, weather forecast, news feed, upcoming appointments, and more.

In this project, we are going to build a gesture-controlled smart mirror with swipeable pages.

Cutting the Wood

For the rigid construction of the mirror frame, we are using an 18 mm MDF board.

Then only the frame can withstand our huge monitor.

These are the reasons why we considered the MDF board in this project.

• MDF is hard to both flex or crack.
• MDF is more affordable and easier to supply
• MDF is easier to paint and seal.

We can cut the wood to the required length by taking proper measurements.

Here we used Jigsaw for cutting the wood, it is the only power tool with us, you can also use the table saw or circular saw instead. For long straight cuts with the jigsaw use straightedge or guides. To get a smoother finish on cutting use a blade that is designed to cut metal. Here front frame and back frame constitute the total frame.

Miter Joints

For the front frame, we are going for miter cuts, so we can roughly mark the positions and cut down the wood.

Miter joints can give a nice and clean appearance to the frame.

Assembling The Frame

We assembled the frame by adding some synthetic resin adhesives which will take a long time to cure. For creating more inward pressure we have used right angled clamps and left it over tonight, don't give excess clamp pressure which may lead to poor binding.

Here is what it looks like.

So we can add the wooden pieces which are left over earlier to the front frame using instant glues.

You should be careful while using these types of glue, they will cure at lightning speed, but the final strength is only reached after about 24 hours.

Adding Some Screws

To give extra rigidity to the frame, we added some screws to the front frame after pre-drilling pilot holes. Screws are lubricated with bar soap for making the process easier.

We also tightened the back frame using the same screws.

Sanding and Painting

Sanding is done to create an even surface and enhance the look of wood. It can take time to do a good job of sanding, but every project benefits from sanding. So we are started with fine grits and ended with very fine grits.

Then we primed the surface.

After that, we stained the frame with dark brown color which can give much more contrast to the wall.

Putting Mirror

So we have our two-way glass right here from the local store. we have bought it for 2$, and it was very cheap.

It works on a principle of light intensity. As you can see, If the light intensity is the same on both sides of the glass, the mirror will act like a normal piece of glass, you can see my hand moving. When we placed the mirror on a black sheet, the light is bright on one side and much darker on the other, then the glass will look like a mirror to the people on the brighter side.

Now we can put the mirror in.

There exist some tiny space between the mirror and the frame, we covered it with hot glues. Before proceeding into further steps we cleaned the backside and front side of the mirror with glass cleaners.

Placing Monitor and Essential Components

For putting the monitor behind the mirror, we need to remove the stand of the monitor. Then we carefully inserted the monitor.

Using a Raspberry Pi is an easy way to make a smart mirror, as they're open-source and very versatile. We have already configured our Raspberry Pi 4. The monitor only supports VGA cable, already we have a micro HDMI to HDMI cable, so we arranged an HDMI to VGA converter. Now we can display our raspberry pi with the monitor.

So we secured the RPI on the monitor with double-sided tapes and connected all the cables. We also attached the camera module and wired it to the raspberry pi.

To secure the monitor in the position we added a wooden block to the back of the frame.

There will be two power cables coming out from the mirror, one is for Monitor and another is for raspberry pi. Then we booted up the raspberry pi.

Magic Mirror

MagicMirror² is an open-source modular smart mirror platform. With a growing list of installable modules, the MagicMirror² allows you to convert your hallway or bathroom mirror into your personal assistant.

MagicMirror² is developed to run on a Raspberry Pi. It might (and will) run on different types of hardware, but new versions will only be tested on a Raspberry Pi.

Electron, the app wrapper around MagicMirror², only supports the Raspberry Pi 2, 3 & 4. The Raspberry Pi 0/1 is currently not supported. If you want to run this on a Raspberry Pi 1, use the server-only feature and setup a fullscreen browser yourself.

Install Magic Mirror

1. Download and install the latest Node.js version:

• curl -sL https://deb.nodesource.com/setup_16.x | sudo -E bash -
• sudo apt install -y nodejs

2. Clone the repository and check out the master branch: git clonehttps://github.com/MichMich/MagicMirror

3. Enter the repository: cd MagicMirror/

4. Install the application: npm install

5. Make a copy of the config-sample file: cp config/config.js.sample config/config.js

6. Start the application: npm run start For Server Only use: npm run server

Additional Modules

1. MMM-Carousel

This is an extension to the MagicMirror² project, allowing the modules to be displayed in a rotating carousel instead of displaying all of them at once.

There are three modes available:

• 'global' - All modules not cited in the ignoreModules config are rotated, displaying only one at a time for the duration of transitionInterval. This is particularly useful on small screens where there may not be enough space to display several components at once.
• 'positional' - Modules are grouped by position setting and rotated within a position except for modules listed in that position's ignoreModules, an overrideTransitionInterval can also be set to rotate different positions at different speeds.
• 'slides' - groups of modules can be assigned to be displayed at the same time (regardless of position), an unlimited number of these "slide" groups can be set up. Module positions and appearances can also be changed for each slide.
• Slide Indicators and Navigation Buttons can be enabled (see config settings)
• Modules can be moved to different positions and CSS classes applied to them for each slide.
• Multiple instances of a module can be used on different slides.
• Integration with MMM-KeyBindings for keyboard and Bluetooth remote navigation.

Install MMM-Carousel module using the following code

cd ~/MagicMirror/modules
git clone https://github.com/shbatm/MMM-Carousel

2. MMM-KeyBindings

The MMM-KeyBindings Module is a helper module that provides a method for controlling the MagicMirror² through a Bluetooth-connected Remote or through a keyboard by capturing button presses and keystrokes and either processing them or passing them on for use in other modules via notifications.

The primary features are:

• Control from Amazon Fire Stick remote control or other BlueTooth devices.
• Customizable keyboard navigation and control.
• Basic navigation keys are captured, but this can be changed in the config.
• Key Presses are sent to other modules for action via notification.
• Assign keys to perform certain actions automatically (e.g. turn on/off the monitor when the HOME key is pressed, using MMM-Remote-Control).
• Allows a module to "take focus", allowing other modules to ignore keypresses when a particular module has focus (e.g. in a pop-up menu).
• Allows for multiple instances of the MagicMirror to be open on different screens and be independently controlled.

Install MMM-KeyBindings module using the following code

cd ~/MagicMirror/modules
git clone https://github.com/shbatm/MMM-KeyBindings

3. MMM-Covid19

This is a module designed to display the Covid-19 cases as a chart.

Install the MMM-Covid19 module using the following code.

cd ~/MagicMirror/modules
git clone https://github.com/SimHub/MMM-Covid19

Edit Config File

After installing the required modules edit the config file which is available at ~MagicMirror/config/config.js.

Download the config file from here and replace the config file.

Gesture Recognition Using MediaPipe

MediaPipe Hands is a high-fidelity hand and finger tracking solution. It employs machine learning (ML) to infer 21 3D landmarks of a hand from just a single frame. Whereas current state-of-the-art approaches rely primarily on powerful desktop environments for inference, our method achieves real-time performance on a mobile phone, and even scales to multiple hands. We hope that providing this hand perception functionality to the wider research and development community will result in an emergence of creative use cases, stimulating new applications and new research avenues.

ML Pipeline

MediaPipe Hands utilizes an ML pipeline consisting of multiple models working together: A palm detection model that operates on the full image and returns an oriented hand bounding box. A hand landmark model that operates on the cropped image region defined by the palm detector and returns high-fidelity 3D hand keypoints. This strategy is similar to that employed in our MediaPipe Face Mesh solution, which uses a face detector together with a face landmark model.

Providing the accurately cropped hand image to the hand landmark model drastically reduces the need for data augmentation (e.g. rotations, translation, and scale) and instead allows the network to dedicate most of its capacity towards coordinate prediction accuracy. In addition, in our pipeline, the crops can also be generated based on the hand landmarks identified in the previous frame, and only when the landmark model could no longer identify hand presence is palm detection invoked to relocalize the hand.

The pipeline is implemented as a MediaPipe graph that uses a hand landmark tracking subgraph from the hand landmark module, and renders using a dedicated hand renderer subgraph. The hand landmark tracking subgraph internally uses a hand landmark subgraph from the same module and a palm detection subgraph from the palm detection module.

Models

Palm Detection Model

To detect initial hand locations, we designed a single-shot detector model optimized for mobile real-time uses in a manner similar to the face detection model in MediaPipe Face Mesh. Detecting hands is a decidedly complex task: our lite model and full model have to work across a variety of hand sizes with a large scale span (~20x) relative to the image frame and be able to detect occluded and self-occluded hands. Whereas faces have high contrast patterns, e.g., in the eye and mouth region, the lack of such features in hands makes it comparatively difficult to detect them reliably from their visual features alone. Instead, providing additional contexts, like arm, body, or person features, aids accurate hand localization.

Our method addresses the above challenges using different strategies. First, we train a palm detector instead of a hand detector, since estimating bounding boxes of rigid objects like palms and fists is significantly simpler than detecting hands with articulated fingers. In addition, as palms are smaller objects, the non-maximum suppression algorithm works well even for two-hand self-occlusion cases, like handshakes. Moreover, palms can be modeled using square bounding boxes (anchors in ML terminology) ignoring other aspect ratios, and therefore reducing the number of anchors by a factor of 3-5. Second, an encoder-decoder feature extractor is used for bigger scene context-awareness even for small objects (similar to the RetinaNet approach). Lastly, we minimize the focal loss during training to support a large number of anchors resulting from the high scale variance.

With the above techniques, we achieve an average precision of 95.7% in palm detection. Using a regular cross-entropy loss and no decoder gives a baseline of just 86.22%.

Hand Landmark Model

After the palm detection over the whole image our subsequent hand landmark model performs precise keypoint localization of 21 3D hand-knuckle coordinates inside the detected hand regions via regression, that is direct coordinate prediction. The model learns a consistent internal hand pose representation and is robust even to partially visible hands and self-occlusions.

To obtain ground truth data, we have manually annotated ~30K real-world images with 21 3D coordinates, as shown below (we take Z-value from the image depth map if it exists per corresponding coordinate). To better cover the possible hand poses and provide additional supervision on the nature of hand geometry, we also render a high-quality synthetic hand model over various backgrounds and map it to the corresponding 3D coordinates.

Python Code

Install the MediaPipe python package using pip

pip install mediapipe

Now using the following code make sure the code detects the gestures.

import cv2
import mediapipe as mp
import pyautogui

mp_drawing = mp.solutions.drawing_utils
mp_hands = mp.solutions.hands
##################################
tipIds = [4, 8, 12, 16, 20]
state = None
Gesture = None
wCam, hCam = 720, 640
############################
def fingerPosition(image, handNo=0):
    lmList = []
    if results.multi_hand_landmarks:
        myHand = results.multi_hand_landmarks[handNo]
        for id, lm in enumerate(myHand.landmark):
            # print(id,lm)
            h, w, c = image.shape
            cx, cy = int(lm.x * w), int(lm.y * h)
            lmList.append([id, cx, cy])
    return lmList
# For webcam input:
cap = cv2.VideoCapture(0)
cap.set(3, wCam)
cap.set(4, hCam)
with mp_hands.Hands(
    min_detection_confidence=0.8,
    min_tracking_confidence=0.5) as hands:
  while cap.isOpened():
    success, image = cap.read()
    if not success:
        print("Ignoring empty camera frame.")
      # If loading a video, use 'break' instead of 'continue'.
        continue
    # Flip the image horizontally for a later selfie-view display, and convert
    # the BGR image to RGB.
    image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
    image.flags.writeable = False
    results = hands.process(image)
    # Draw the hand annotations on the image.
    image.flags.writeable = True
    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
    if results.multi_hand_landmarks:
      for hand_landmarks in results.multi_hand_landmarks:
        mp_drawing.draw_landmarks(
            image, hand_landmarks, mp_hands.HAND_CONNECTIONS)
    lmList = fingerPosition(image)
    #print(lmList)
    if len(lmList) != 0:
        fingers = []
        for id in range(1, 5):
            if lmList[tipIds[id]][2] < lmList[tipIds[id] - 2][2]:
                #state = "Play"
                fingers.append(1)
            if (lmList[tipIds[id]][2] > lmList[tipIds[id] - 2][2] ):
               # state = "Pause"
               # pyautogui.press('space')
               # print("Space")
                fingers.append(0)
        totalFingers = fingers.count(1)
        print(totalFingers)
        #print(lmList[9][2])

        if totalFingers == 4:
            state = "Play"
           # fingers.append(1)
        if totalFingers == 0 and state == "Play":
            state = "Pause"
            pyautogui.press('space')
            print("Space")
        if totalFingers == 1:
            if lmList[8][1]<300:
                print("left")
                pyautogui.press('left')
            if lmList[8][1]>400:
                print("Right")
                pyautogui.press('Right')
        if totalFingers == 2:
            if lmList[9][2] < 210:
                print("Up")
                pyautogui.press('Up')
            if lmList[9][2] > 230:
                print("Down")
                pyautogui.press('Down')
    #cv2.putText(image, str("Gesture"), (10,40), cv2.FONT_HERSHEY_SIMPLEX,
     #              1, (255, 0, 0), 2)
    cv2.imshow("Media Controller", image)
    key = cv2.waitKey(1) & 0xFF
    # if the `q` key was pressed, break from the loop
    if key == ord("q"):
        break
  cv2.destroyAllWindows()

Working

1. Start the gesture detection code(cloned from the Github Repo) using the following command.

python finger.py

2. Start the MagicMirror using the following command.

cd ~/MagicMirror
npm run start

3. Now swipe in front of the camera and see your pages flipping.