Benchmarking TensorFlow Lite on Raspberry Pi

INTRO

This project documents setting up and running the benchmarks on the Raspberry Pi. I created this project as part of my efforts to benchmark Tensorflow Lite on the MaaXBoard. I wanted to compare the MaaXBoard's performance running inference to similar developer boards, so I chose the Google Coral and Raspberry Pi 3 Model B+.

• Instructions for benchmarking the MaaXBoard are here.
• Instructions for benchmarking the Coral dev board are here.
• Final benchmarking article is here.
• Alasdair Allan's original big benchmarking roundup is here.

Alasdair already created a great guide to benchmarking the Raspberry Pi here. A few things have changed with the install since then because of updates to Tensorflow Lite and OpenCV support on Rasbperry Pi, so this is basically an updated version of that guide, plus a few extras.

SETUP

I'll be running the benchmarks on my pi via SSH. Make sure SSH is enabled on your pi. If your pi is connected to WIFI you can get the IP address by typing "ping raspberrypi.local" in your host computer's terminal. Now ssh into your pi:

ssh pi@[IP ADDRESS]

Your Raspberry pi should come with Python installed. Check which version you have:

python3 --version

My result: Python 3.7.3

Install Tensorflow Lite

It's only one line to install Tensorflow Lite on Raspberry Pi using pip. Make sure the python tag is "cp37" for Python version 3.7, and the other tags are linux_armv71.

pip3 install https://dl.google.com/coral/python/tflite_runtime-2.1.0.post1-cp37-cp37m-linux_armv7l.whl

Install OpenCV

OpenCV is used in the benchmark script as a tool for reading and resizing the images.

OpenCV requires a lot of other modules for the install. This helpful blog post by Emmet on pimylifeup.com details what each module does. I think this is useful information, so I've detailed it in the instructions below:

• This command will install the packages that contain the tools needed to compile the OpenCV code.

sudo apt install cmake build-essential pkg-config git

• Install the packages that will add support for different image and video formats to OpenCV.

sudo apt install libjpeg-dev libtiff-dev libjasper-dev libpng-dev libwebp-dev libopenexr-dev 
sudo apt install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libx264-dev libdc1394-22-dev libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev

• Next, install all the packages needed by OpenCV's interface:

sudo apt install libgtk-3-dev libqtgui4 libqtwebkit4 libqt4-test python3-pyqt5

• These packages are crucial for getting OpenCV to run at a decent speed on rpi:

sudo apt install libatlas-base-dev liblapacke-dev gfortran

• The second last lot of packages thaat we need to install relate to the Hierarchical Data Format (HDF5) that OpenCV uses to manage data.

sudo apt install libhdf5-dev libhdf5-103

• Install the final few packages by using the command below:

sudo apt install python3-dev python3-pip python3-numpy

• Finally, install OpenCV:

pip3 install opencv-python

Now test your OpenCV install:

python3 -c 'import cv2; print(cv2.__version__)'

I got the error:

undefined symbol: __atomic_fetch_add_8

Apparently this is an issue as of Raspbian Buster. Thanks to pyimagesearch, I was able to solve it.

Readers have reported that some versions of OpenCV 4 as installed via pip do not work properly on the Raspberry Pi. You may encounter an `“undefined symbol: __atomic_fetch_add8″` forlibatomicerror when youimport cv2from Python if you do not use the specific version of OpenCV mentioned in the code block above.

The solution:

pip3 install opencv-contrib-python==4.1.0.25

Copy and unzip the benchmarking files:

The original files Alasdair created were hosted on dropbox here. Because of changes to Tensorflow v.2, I've updated the code and also included code to benchmark image classification. The only thing it doesn't contain is the EfficientNet-Lite model, but that we can download from the Google Coral github.

I've attached the necessary models and scripts in the code section of this project. You can download them to the Raspberry Pi like this:

curl -L https://hacksterio.s3.amazonaws.com/uploads/attachments/1143280/RASPBERRYPI.zip > files.zip
unzip files.zip
cd RASPBERRYPI

=====RUN THE BENCHMARKS=====

Run the benchmark for MobileNet v.2 on Raspberry Pi:

python3 benchmark_tf_lite.py --model tflite_for_rpi_V2/mobilenet_v2.tflite --label tflite_for_rpi_V2/coco_labels.txt --input fruit.jpg --output out.jpg --runs 1000

Elapsed time is 1108.495098892 ms

apple score = 0.953125

banana score = 0.83984375

Run the benchmark for MobileNet v.1 on Raspberry Pi:

python3 benchmark_tf_lite.py --model tflite_for_rpi_V1/mobilenet_v1.tflite --label tflite_for_rpi_V1/coco_labels.txt --input fruit.jpg --output out.jpg --runs 1000

Elapsed time is 815.3825484200001 ms

apple score = 0.80078125

apple score = 0.515625

Neither the speed nor the accuracy is great here. As you can see, the image detector thinks the banana is an apple:

Run EfficientNet

Download model from here: https://coral.ai/models/

I downloaded it to my host computer and copied it over to the board:

scp efficientnet-edgetpu-L_quant.tflite pi@[IP ADDRESS]:

Get labels:

curl -OL https://raw.githubusercontent.com/google-coral/edgetpu/master/test_data/imagenet_labels.txt > imagenet_labels.txt

Benchmark for EfficientNet L on Raspberry Pi:

python3 benchmark_tf_lite.py --model efficientnet-edgetpu-L_quant.tflite --label imagenet_labels.txt --input fruit.jpg --runs 1000

This is painfully slow. I'd recommend going and doing something else while this runs.

Elapsed time is 8143.91 ms

0.5340: honeycomb

0.2453: screwdriver

0.0374: nail

Benchmark the lite0 model

I also tried a different model, the EfficientNet-Lite lite0 model, which IS optimized for CPU. It's quite a bit smaller than the other version - only 5.17MB: https://tfhub.dev/tensorflow/lite-model/efficientnet/lite0/int8/1

Benchmark for EfficientNet L on Raspberry Pi:

python3 benchmark_classification.py --model efficientnet_lite0_int8_1.tflite --label imagenet_labels.txt --input fruit.jpg --runs 1000

This one is much much faster and way more accurate!

Elapsed time is 638.5257380580006 ms

0.125490: banana

0.043137: honeycomb

0.027451: cauliflower

FINAL RESULTS

Here's a quick preview of speed vs price for the Google Coral and its competitors. If you want to know how the MaaXBoard and Coral Dev Board performed, the benchmarking article is here.