Smarter Vector Robot

Part 1: Getting started

1.1 Download and flash PYNQ 2.5 on the SD card: http://avnet.me/ultra96v2-pynq-image-v2.5 I used balenaEtcher.

1.2 Turn on the Ultra96v2 board and connect it to host computer using a USB mini cable.

• insert miniSD card into SD slot
• connect miniUSB cable to host computer
• connect power
• press power button (first blue button in the right of the board)
• wait to power on and heartbeat led to flash (4th led between the USB ports)

• connect to Jupyter Notebooks running on the Ultra96v2 board by opening http://192.168.3.1:9090/ in a browser
• ssh into Ultra96v2 board and configure wifi network (user: xilinx pass: xilinx)

$ ssh xilinx@192.168.3.1
$ sudo nano /etc/netplan/50-cloud-init.yaml

The file should look like:

network:
    ethernets:
        eth0:
            dhcp4: true
            optional: true
            version: 2
    wifis:
        wlan0:
            dhcp4: true
            optional: true
            access-points:
                "SSID_name":
                    password: "WiFi_password"

Save and run:

$ sudo netplan generate
$ sudo netplan apply

This will start and connect to wifi at boot time.

Alternatively install remote desktop software or use a miniDP-DP cable or miniDP-HDMI cable. Not all adapters work.

$ sudo apt install xrdp
$ sudo systemctl status xrdp

As an alternative a comprehensive guide (for Ultra96v1 board) can be found here. It can be easily adapted for v2 version of the board.

1.3 Install PYQN-DPU: https://github.com/Xilinx/DPU-PYNQ

Upgrade the PYNQ v2.5 image:

$ sudo su
# git clone --recursive --shallow-submodules https://github.com/Xilinx/DPU-PYNQ.git
# cd DPU-PYNQ/upgrade
# make

Install

# pip3 install pynq-dpu
$ cd $PYNQ_JUPYTER_NOTEBOOKS
$ pynq get-notebooks pynq-dpu -p .

Run the examples from the pynq-dpu folder in Jupyter Notebooks.

1.4 Install the Anki Vector SDK & configure it

$ sudo apt-get install python3-pil.imagetk
$ sudo su
# python3 -m pip install anki_vector
# python3 -m anki_vector.configure

The configuration script will ask for Vector's serial number, IP and will require an account with Anki. Please check the guide how to obtain these. If everything runs correctly the configure script will create two files in /root/.anki_vector/ directory. You can copy these two files in /home/xilinx/.anki_vector/ so the user xilinx can also connect to Vector. Also edit /home/xilinx/.anki_vector/sdk_config.ini to point to the correct.cert file.

Jupyter Notebook run as root on Ultra96v2 board so both root and xilinx users should be able to connect to the Vector robot.

1.5 Get Vector SDK and test connection

$ git clone https://github.com/anki/vector-python-sdk
$ cd vector-python-sdk/examples/tutorials/
$ python3 01_hello_world.py

If everything was configured correctly the robot should say "Hello World!".

Part 2: Test inference

PYNQ-DPU project has a Jupyter notebook example on how to run yolov3 inference. Run it and get intimate with it. We will modify it to run inference on images captured by the Vector's camera. It can be found here: http://192.168.3.1:9090/notebooks/pynq-dpu/dpu_yolo_v3.ipynb

2.1 Duplicate the pynq-dpu/dpu_yolo_v3.ipynb notebook and modify it to capture images from Vector.

Change this part:

With this:

Run the steps. At the end the inference should detect a person in the image:

Part 3: Run yolov3 inference on images captured by Vector's camera

For this we will modify the remote_control example app from Vector SDK examples to run inference on all images captured by Vector. Must be run as root:

# git clone https://github.com/andrei-ace/vector_ultra96v2.git
# cd vector_ultra96v2/remote_control/
# python3 remote_control.py

Open a browser and go to http://192.168.3.1:5000/

Part 4: Demo

Part 5: Manual building and using different model

The DPU can be manually build following the instructions from https://github.com/Xilinx/DPU-PYNQ/blob/master/boards/README.md

and https://github.com/Xilinx/Vitis-AI/blob/v1.1/DPU-TRD/prj/Vitis/README.md

Other models can be found here: https://github.com/Xilinx/AI-Model-Zoo/tree/7f3456b26724cc649960e3b6924488859eebe489

Part 6: Useful links

http://zedboard.org/product/ultra96-and-pynq-framework

https://github.com/Xilinx/DPU-PYNQ

https://github.com/Xilinx/Vitis-AI

http://www.pynq.io/