Panasonic's Video Intercom Hacked to Be Alexa Enabled

This kind of video intercom is very common in typical apartments with a single shared entrance in Japan. Most models don’t have any IoT capabilities, and since they’re integrated into the building security system, it’s virtually impossible to replace them with smart doorbells. So, some people have hacked intercoms in their apartments using ADCs, light sensors, or audio sensors to detect calls and using servo motors to perform unlock button presses. However, since I love hacks that people cannot tell IoT capabilities have been added just by looking from outside, this time I hacked one in my home (Panasonic’s SHVT18612WK), in a way that

• its original functionalities are intact and the hack is not visible from the outside.
• call detections and unlock operations are performed 100% electronically.
• it securely connects to the AWS cloud.
• it plays pre-recorded messages.

Enjoy a video of the hack before and after.

The principle of operation

I used an ESP32 microcontroller with Mongoose OS because I believe this is one of the best combinations for IoT projects. I used optocouplers to electronically detect calls and unlock the entrance door, and DFPlayerMini to play a pre-recorded message.

Electronically detecting calls

When a video call is initiated from the entrance, the key-shaped green LED starts blinking on the video intercom. So, by connecting the primary side of an optocoupler in parallel to this LED and the secondary side to a GPIO and GND, and checking the state of the GPIO, you can detect calls.

When a call is detected, ESP32 publishes a message to AWS IoT Core. AWS Lambda is then automatically executed to send a Slack message.

Electronically performing unlock button presses

By connecting the primary side of an optocoupler to a GPIO and GND, and the secondary side in parallel to the unlock push button on the intercom circuit board, and setting the GPIO from L to H, then to L with slight delays in between, you can perform unlock operations without physically pressing the unlock button. The same is true for the talk button.

An AWS Amplify web app and an Alexa skill publish a message to AWS IoT Core, and ESP32 receives the message and does the job.

Step 1. Expose the main PCB and look for hack spots

First, remove the side covers from the video intercom and loosen screws.

Make sure to turn the power switch off before removing the bundle of wires. Most models come with a built-in fire alarm and it goes off when those wires are removed, cut, or burnt with the power switch on.

Look for spots that can be easily hacked.

Solder a ribbon cable to the hack spots. (cable color)

• Unlock standby LED + (Orange)
• Unlock standby LED – (Red)
• Unlock button (Blue)
• Talk button (Green)
• Unlock/Talk button shared GND (Yellow)

Step 2. Build an electrical circuit

On a mini breadboard, place an ESP32-DevKitC, optocouplers, and a DFPayerMini. To avoid stability issues as much as possible, use solid jumper wires instead of flexible ones.

I found the voltage across the key-shaped LED on the video intercom panel to be approx. 3.0V, so I used IF=7.5mA (and R=250ohm) for the optocoupler for call detections, which is the recommended value for the operation under VF=1.17V according to its datasheet. If this IF is too big, the key-shaped LED won’t blink, and if it’s too small, the optocoupler won’t work. I used IF=15mA (and R=120ohm) for the optocouplers for talk and unlock button presses.

For DFPlayerMini, connect Rx only because for this hack ESP32 doesn’t need to know when it has finished playing audio files, so Tx is not necessary. Use Amazon Polly to generate audio messages in mp3 format and save them to a microSD.

The actual electric circuit looks the following. You could design a PCB for this.

Step 3. Write a code for ESP32 and provision it for AWS IoT

For this project, I used Mongoose OS, a very powerful IoT firmware development framework, so the application code (init.js) can be written in JavaScript. The full code is found in GitHub.

First, install Mongoose OS to ESP32 and connect it to Wi-Fi using the following commands.

$ mos flash esp32
$ mos wifi SSID PASSWORD

Start writing init.js by loading Mongoose OS APIs, and then declare variables.

// Load Mongoose OS APIs
load('api_gpio.js');
load('api_mqtt.js');
load('api_sys.js');
load('api_timer.js');
load('api_uart.js');

// Declare variables
let ledPin = 12;
let talkBtn = 13;
let unlockBtn= 14;
let callState = false;
let uartNo =1;
let topic1 = 'intercom/detect';
let topic2 = 'intercom/unlock';
let qos = 1;

Set up UART for DFPlayerMini.

// UART Setup
UART.setConfig(uartNo, {
  baudRate: 9600,
  esp32: {
    gpio: {
      rx: 25,
      tx: 26,
    },
  },
});

Set GPIO mode and initialize GPIOs.

// Set GPIO mode
GPIO.setup_input(ledPin, GPIO.PULL_UP);  // iput & internally pulled up
GPIO.set_mode(startTalkingBtn, GPIO.MODE_OUTPUT);
GPIO.set_mode(openSecurityDoorBtn, GPIO.MODE_OUTPUT);

// Initialize GPIOs
GPIO.write(startTalkingBtn, 0);
GPIO.write(openSecurityDoorBtn, 0);

The following code block detects calls and publishes a message to topic1. Mongoose OS comes with a useful button handler for detecting button presses. The ledPin is internally pulled up, so it’s normally H and becomes L when a call is initiated. Timer.set() makes callState back to normal after 15 sec.

// Detect calls and publish a message to topic1
GPIO.set_button_handler(ledPin, GPIO.PULL_UP, GPIO.INT_EDGE_NEG, 20, function(x) {
​
  if (!callState) {
​
    callState = true;
    let message = JSON.stringify({ });
    let ok = MQTT.pub(topic1, message, qos);
    print(ok);
    print("-----Call detected, hopefully published to AWS IoT-----");
​
    Timer.set(15000, false, function() {
      callState = false;
      print("-----Back to normal-----");
    }, null);
​
  }
​
}, true);

To send notifications by Slack, you can use messages published to topic1 and Incoming Hooks. Create a Lambda function that sends a message to a Slack channel and configure AWS IoT Rules to trigger this Lambda.

The following code block sequentially performs a talk button press, audio playback, an unlock button press, and a talk button press when a message is received from AWS IoT Core. I used 3 timers here because in Mongoose OS for this kind of use case, it is recommended to use Timer.set() instead of Sys.usleep() for stable operations.

// Subscribe to topic2 and unlock door when message is received
MQTT.sub(topic2, function(conn, msg) {

  print('-----Received message from AWS IoT-----')
  talk();
​
  Timer.set(2000, false, function() {
    play();
  }, null);

  Timer.set(4500, false, function() {
    unlock();
  }, null);
​
  Timer.set(9500, false, function() {
    talk();
  }, null);
​
}, true);

What talk() and unlock() do is to set GPIOs from L to H, and then back to L with Sys.usleep() in between. This can simulate the actual button presses.

// Mimic talk button press
function talk(){
  GPIO.write(talkBtn, 1);
  Sys.usleep(300000);
  GPIO.write(talkBtn, 0);
}

// Mimic unlock button press
function unlock(){
  GPIO.write(unlockBtn, 1);
  Sys.usleep(300000);
  GPIO.write(unlockBtn, 0);
}

DFPlayerMini can be controlled with UART. The reference is here.

// Play /01/001.mp3 with DFPlayerMini
function play(){
  UART.write(uartNo, '\x7E');
  UART.write(uartNo, '\xFF');
  UART.write(uartNo, '\x06');
  UART.write(uartNo, '\x0F');
  UART.write(uartNo, '\x00');
  UART.write(uartNo, '\x01');
  UART.write(uartNo, '\x01');
  UART.write(uartNo, '\xEF');
}

Since Mongoose OS contains the AWS IoT Device SDK for Embedded C and does everything necessary to connect your ESP32 to AWS IoT Core, what you need to do is to execute the following command. Make sure you get your Access Key ID and Secret Access Key pair ready and set up on your computer beforehand.

$ mos aws-iot-setup --aws-region AWS_REGION

For Alexa skill, create one with the welcome intent that publishes a message to topic2, intercom/unlock. If you use python, something like this will work.

client = boto3.client('iot-data', region_name='es-east-1')
response = client.publish(
    topic='intercom/unlock',
    qos=1,
    payload=json.dumps({ })
)

There are a bunch of how to create Alexa skill and Amplify web apps on the Internet, so please refer to those to create your own skill and app.

Step 4. Do electrical work and hide everything in the walls

Remove the base unit of the video intercom from the wall.

Remove the AC power cords, and make a branch in the AC power line. Attach a wall outlet with USB charging ports to the branch, and the AC power cords back to the base unit.

Use a micro USB cable to power ESP32 and hide everything in the walls. Make sure to place the speaker close to the microphone, otherwise, guests cannot hear the audio message.

Now the hack is done!

Future Directions

I want to make a PCB for this project. I also want to analyze the video signal, take it out from the board, and send it to the AWS Cloud to do fun stuff.