Benchmarking Tensorflow Lite on the Coral Dev Board

INTRO

I created this project as part of my efforts to benchmark Tensorflow Lite on the MaaXBoard. I wanted to compare the MaaXBoard's performance running inference to similar developer boards, so I chose the Google Coral and Raspberry Pi 3 Model B+.

• Instructions for benchmarking the Raspberry Pi 3 Model B+ are here.
• Instructions for benchmarking the MaaXBoard are here.
• Final benchmarking article is here.
• Alasdair Allan's original big benchmarking roundup is here.

In addition to running the benchmark, this guide walks you through getting started with Google Coral. There are already a couple great setup guides for the Coral dev board, including the official getting started guide at coral.ai and Alasdair's guide on Hackster. However, I ran into several problems while using OSX Catalina to connect to the dev board, so I figured I should document them here so other people won't spend hours looking for a workaround like I did.

This guide is specifically for Mac, so if you're on Linux, you should probably follow one of the guides I just mentioned. Windows isn't compatible at all. If you've already set up your Coral dev board before, skip to step 6.

THE BENCHMARK

The Coral dev board is picky about the models it can run. It can only run Tensorflow Lite models that have been pre-compiled for the edge TPU.

MobileNet V.1 and V.2

Luckily for us, there are pre-converted models that meet all of the requirements, like these versions of MobileNet v1 and MobileNet v2. Even luckier - Alasdair Allan created a script to benchmark MobileNet last year, so I can run the exact same script and compare my results to his.

EfficientNet-Lite

I wanted to see what the Coral dev board was capable of, so I chose one more model to benchmark - EfficientNet-Lite.

EfficientNet-Lite is a version of the EfficientNet model that was created by Google for use on edge / mobile. It's specifically tuned for Google Coral's TPU, so I wanted to try that as well to see what kinds of speeds and accuracy I could achieve.

1. SETUP YOUR MACBOOK

The Coral dev board doesn't have a GUI. In order to work with it, you'll have to install a couple tools to connect to it.

1.a Install Screen

Screen is the recommended serial console program. It can be installed with brew. Check t

brew update
brew install screen
screen -v

1.b Install Fastboot

The fastboot protocol is a mechanism for communicating with bootloaders over USB or ethernet. It is designed to be very straightforward to implement, to allow it to be used across a wide range of devices and from hosts running Linux, macOS, or Windows.

Download Google's Platform-Tools (includes fastboot): Mac

Google Corals instructions say to do it like this:

mkdir -p ~/.local/bin
sudo mv ~/Downloads/platform-tools/fastboot ~/.local/bin/

However, because Catalina blocks 3rd party developers, it has to be done like this:

brew cask install android-platform-tools
fastboot --version
fastboot version 29.0.6-6198805
Installed as /usr/local/bin/fastboot

It will give you a warning, so go to under Security and Privacy settings, enable this 3rd party developer, and then click "open" when the popup returns.

Check that it's installed:

fastboot --version

For Mac compatibility, the version must be 28.0.2 or higher.

1.c Install Mendel Developer tool on your Mac

The Mendel developer tool (mdt) is supposed to let you transfer an SSH key to the board. You can install it with pip.

$ pip3 install --user mendel-development-tool

As you'll see later, mdt doesn't work correctly on OSX Catalina, so the workaround is detailed in step 5.

On mac, Mendel requires Silicon Lab's CP210x USB to UART Bridge Virtual COM Port (VCP) driver for Mac. I believe that the problem is with the driver. Hopefully they update it soon! Maybe they already have. Install the latest version just in case.

2. FLASH THE BOARD

Now is the moment you've been waiting for! You get to use your board.

First, check to make sure your boot switches are set to "on, off, off, off" as pictured:

Connect micro-USB to your board, and the other end to your computer.

2.a Initiate fastboot mode

Checkyour serial number. If your serial number starts with "9410" or higher, then it was manufactured on April 10th, 2019, or later, so you can skip this step and go to step 2.b (because your board automatically boots into fastboot mode).

[PHOTO OF SERIAL #S]

In a terminal on your mac, use screen to connect to the board.

screen /dev/cu.SLAB_USBtoUART 115200

Immediately plug the board into power (power is the USB-C port closest to the audio input).

You should see something like this after it starts up (if you don't, press enter):

**********************************************************************
Welcome to your new Coral EdgeTPU Development Board!
**********************************************************************
To get started, you need to download and flash the latest firmware.
Please follow the instructions at g.co/coral/setup.

Then type:

fastboot 0

The board is now in fastboot mode

2.b Execute the flash script

Plug another USB-C cable into the board's data port (USB-C closest to HDMI) Open a new terminal window (ctrl-n) and run "fastboot devices"

fastboot devices

output: 170a99d6ef944d3f fastboot

cd ~/Downloads

curl -O https://dl.google.com/coral/mendel/enterprise/mendel-enterprise-day-13.zip

unzip mendel-enterprise-day-13.zip && cd mendel-enterprise-day-13

bash flash.sh

Output: Sending 'bootloader0' (991 KB) OKAY [ 0.057s]

etc..

Once flashing is done, shut down the Google Coral:

sudo shutdown now

Then unplug your USB-C power cable and USB-C data cable.

NOTE: When you're done using the Coral dev board, always use this command to shut the board down. Don't simply unplug the board without shutting it down first, since that could corrupt the image.

3. CONNECT TO WIFI ON YOUR BOARD

With the USB-B pluggd in, from terminal, run:

screen /dev/cu.SLAB_USBtoUART 115200

Then immediately plug a micro-B USB cable into the MaaXboard's micro-USB port. The system of the dev board will start loading, the login prompt will come up. Password and username are both "mendel."

3.a Use nmcli to connect to your wifi

nmcli dev wifi connect 'YOUR SSID' password 'YOUR PASSWORD' ifname wlan0

Device 'wlan0' successfully activated with 'c473dd7c-e7ca-4827-821f-efdaa893cc6d'.

Note: dual band routers won't work.

3.b Make it more permanent by changing the etc/hosts file

sudo nano /etc/wpa_supplicant/wpa_supplicant.conf

ctrl_interface=DIR=/var/run/wpa_supplicant

GROUP=wheel network={

ssid="MermaidPalace2.4"

scan_ssid=1

key_mgmt=WPA-PSK

psk="aquarius"

}

You can certify the connection with

nmcli connection show

Or with

nmcli dev wifi

(connected network will have a * next to it):

4. UPDATE THE SOFTWARE

sudo apt-get update
sudo apt-get dist-upgrade

5. CONNECT TO YOUR BOARD VIA MDT

5.a Connect to the board's shell via mdt

mdt is supposed to be a user-friendly version of SSH that will automatically generate a public key and share it with your PC over USB. Other ways of connecting to the board, like regular ssh, are disabled, so if mdt isn't working it makes it difficult to share files, etc with the board from your mac. Read on to learn about the problem I ran into and how I re-enabled SSH.

At this point, Google's instructions say to plug the USB-C cable into the data port (the USB-C port closest to the HDMI port) and unplug the USB-B cable. On a Linux machine, and on older versions of Mac, the UART device shows up as a network connection. That's what the Silicon labs driver you installed earlier is intended to help with. Open a new terminal and run:

mdt shell

If you're able to connect to your device, congrats! The issue has been fixed. Skip to step 6. If not, you'll get the error:

Unable to find any devices on your local network segment.

This is a problem on macOS Catalina. In spite of having the Silabs driver installed, it doesn't "see" USB connection as a network connection. Mdt is only able to see the Coral dev board wifi, but when you try to connect via WIFI using mdt (mdt shell [WIFI IP]), it won't exchange keys, and gives the message "Cowardly refusing to attempt to push a key to a public machine."

To see what was wrong with the USB connection, I opened a new terminal and ran:

sudo ifconfig -a

I realized that the USB to UART actually shows up as two different devices with two different IP addresses.

As you can see in the above screenshot, my USB1's IP address is 192.168.101.2, and USB2's IP address is 192.168.100.2, but running the command mdt shell with the explicit IP address can't connect to either of them:

Note - if you run mdt and get the error "command not found: mdt" make sure mdt is added to your path.

THE FIX FOR OSX CATALINA

I finally found part of the fix in this blog post and the rest of the fix in this video:

Try connecting to your Coral Dev Board using the WIFI IP address you found earlier (mine was 192.168.1.28):

mdt shell 192.168.1.28

You'll get this error:

"Cowardly refusing to attempt to push a key to a public machine."

In another terminal window on Mac, generate a new key:

ssh-keygen

save its as ./mdt.key

Copy your key to the mdt config on your macbook:

cp mdt.key ~/.config/mdt/keys

View your recently created public key:

cat mdt.key.pub

If you're still connect to the Coral dev board via screen, open that terminal, otherwise reconnect with screen /dev/cu.SLAB_USBtoUART 115200 (you may need to unplug and re-plug USB-B, unplug the data USB-C if it's plugged in, and don't forget to press "enter").

Connected to the Coral dev board. In the "mendel" directory, type:

mkdir .ssh
cd .ssh

Create a new file within the.ssh folder called authorized_keys:

nano authorized_keys

Paste in the public key you just created with ssh_keygen and save the file with ctrl-X (or however you like to do it). Get your board's IP address:

ip addr | grep wlan0

Now if you type:

ssh mendel@192.168.1.28

You should be able to SSH into the board.

5.b Edit the Google Coral's ssh_config:

The above fix depends on your Coral dev board's IP address in /.ssh/known_hosts staying the same. Every time your board's IP address changes you'll have to create a fresh public key, so to make it easier for you to connect, it's easiest to edit the dev board's ssh_config file.

(DISCLAIMER: note that allowing ssh with only password, as I am describing here, is not as safe as using keys, if you do it, please be aware of the risks in case you have sensible data or security is an issue for you.)

In your screen terminal:

• Open ssh_config sudonano /etc/ssh/sshd_config
• In the file, change PermitRootLogin no to PermitRootLogin yes
• Then change PasswordAuthentication no to PasswordAuthentication yes
• Restart the ssh demon with sudo service ssh restart

Now you should be able to copy files back and forth using scp.

Note: if you ever upgrade your board's distro, you'll be asked whether or not you want to upgrade sshd_config. Select "keep the local version currently installed."

6. INSTALL TENSORFLOW LITE

Get your python 3 version on the dev board:

python3 --version

My Python version is 3.7 so I chose the cp37 version for aarch64 linux.

pip3 install https://dl.google.com/coral/python/tflite_runtime-2.1.0.post1-cp37-cp37m-linux_aarch64.whl

========RUN THE BENCHMARKS========

7. BENCHMARK MOBILENET

Ok, all of that was leading up to running our benchmarks. The first benchmark we'll run is MobileNet.

7.a Download the files

The original files Alasdair created were hosted on dropbox here. Because of changes to Tensorflow v.2, I've updated benchmark_edgetpu.py. It's in the code files for this project. The project code also includes two other benchmarking scripts. The only thing it doesn't contain is the EfficientNet-Lite model, but that we can download from the Google Coral github.

You can download the files from Hackster with curl:

curl -L https://hacksterio.s3.amazonaws.com/uploads/attachments/1143279/CORAL.zip > files.zip
unzip files.zip
cd CORAL

7.b Run benchmark_edgetpu.py

Benchmark for MobileNet SSD v.2 on Coral dev board:

python3 benchmark_edgetpu.py \
--model ssd_mobilenet_v2/tflite_for_edgetpu/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite \
--label ssd_mobilenet_v2/tflite_for_edgetpu/coco_labels_for_edgetpu.txt  \
--input fruit.jpg \
--output out.jpg \
--runs 1000

Result:

Elapsed time is 25.81631173300002 ms

apple score = 0.98046875

banana score = 0.93359375

Benchmark for mobilenet SSD v.1 on Coral dev board:

python3 benchmark_edgetpu.py \
--model ssd_mobilenet_v1/tflite_for_edgetpu/tflite_graph_1556320556_edgetpu.tflite \
--label ssd_mobilenet_v1/tflite_for_edgetpu/coco_labels_for_edgetpu.txt  \
--input fruit.jpg \
--output out.jpg \
--runs 1000

Result:

Elapsed time is 18.869002893000015 ms

apple score = 0.7890625

banana score = 0.69921875

dining table score = 0.53125

Wow. Not only does inference correctly classify the apple and banana, it also correctly recognizes that they are sitting on a table!

7.c Running the MaaXBoard's benchmark on Coral

When benchmarking, it's sometimes hard to balance between keeping all things equal, and trying to make the benchmarks imitate actual use cases. Because the edgetpu is able to use the edgetpu detection engine, benchmark_egetpu.py makes use of it. This makes it slightly different from the benchmarking code for the Raspberry Pi and MaaXBoard, which I decided to try running their benchmarking code on MaaXBoard.

7.d Install OpenCV on Coral dev board

The other platforms use OpenCV to resize images, so I decided to try using OpenCV on the Coral DevBoard too. Thankfully, Coral dev board now supports OpenCV.

• Install OpenCV

sudo apt install python3-opencv

• Test installation

python3 -c 'import cv2; print(cv2.__version__)'

7.e Run benchmark_coral.py

Benchmark for Mobilenet SSD v.2 on Coral dev board:

python3 benchmark_coral.py --model tflite_for_edgetpu/mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite --label tflite_for_edgetpu/coco_labels_for_edgetpu.txt --input fruit.jpg --output out.jpg --runs 1000

Result:

Elapsed time is 15.911620770000352 ms

apple score = 0.953125

banana score = 0.83984375

Benchmark for mobilenet SSD v.1 on Coral dev board:

python3 benchmark_coral.py --model tflite_for_edgetpuv1/tflite_graph_1556320556_edgetpu.tflite --label tflite_for_edgetpuv1/coco_labels_for_edgetpu.txt --input fruit.jpg --output out.jpg --runs 1000

Result:

Elapsed time is 12.535867665999831 ms

apple score = 0.80078125

apple score = 0.515625

Copy the file off your Coral dev board so that you can check it out:

scp mendel@[IP ADDRESS]:CORAL/out.jpg .

This version was even faster than the original version. However, the accuracy was much worse, especially for MobileNet v.1. As you can see, mobilenet v.1 misclassified the banana as an apple:

8. BENCHMARK EFFICIENTNET-LITE

EfficientNet-Lite was released in March. It's a smaller version of the full EfficientNet, which was published a year ago. EfficientNet-Lite is an image classification model aimed at mobile applications, and specifically tailored to the edge TPU found on the Coral dev board (and many Android phones).

You can learn more about the Edge TPU's image classification inference engine here. Check out the example code here.

Get the model. There's a 300px version that I used for this benchmark. The "L" stands for large, as in image size. There are three different versions of this model available in the Google Coral model zoo that take 3 different image sizes:

curl -OL https://github.com/google-coral/edgetpu/raw/master/test_data/efficientnet-edgetpu-S_quant_edgetpu.tflite > efficientnet-edgetpu-S_quant_edgetpu.tflite

The corresponding imagenet_labels are in the zip folder provided under the code section (ImageNetLabels.txt).

8.a Run benchmark_coral_efficient.py

Benchmark_coral_efficient.py is the same benchmark as the previous ones, except that it is written to do classification with the Classification Engine.

Benchmark the TPU optimized model

python3 benchmark_coral_efficient.py --model efficientnet-edgetpu/efficientnet-edgetpu-L_quant_edgetpu.tflite --label efficientnet-edgetpu/ImageNetLabels.txt --input fruit.jpg --runs 1000

Result:

Elapsed time is 36.041603453999414 ms

hook, claw score = 0.2421875

banana score = 0.22265625

screwdriver score = 0.06640625

The accuracy is not perfect (banana comes in 2nd with a score of 0.22) but it's blazing fast for a classifier that can recognize 1, 000 images.

Benchmark the lite0 model

I also discovered that there is a version of EfficientNet-Lite that's not specialized for the edge TPU, so I decided to try running this one to see how it performed. It's quite a bit smaller than the other version - only 5.17MB: https://tfhub.dev/tensorflow/lite-model/efficientnet/lite0/int8/1

python3 benchmark_coral_efficient.py --model efficientnet-edgetpu/efficientnet_lite0_int8_1.tflite --label ImageNetLabels.txt --input fruit.jpg --runs 1000

The results from this version are actually quite a bit slower, in spite of the model being smaller. However, it's more accurate! The only thing it sees in this image is a banana:

Elapsed time is 180.288102787 ms

banana score = 0.72265625

Final Results

Here's a quick preview of speed vs price for the Google Coral and its competitors. If you want to know how the MaaXBoard and Rasbperry Pi performed, the full benchmarking article is here.