Face tracking using Kria KR260 Robotics Starter Kit

In this project, we aimed to implement a system which can track a person when they move around in the frame or tries to leave the frame. Usually, there is a tradeoff between the camera's distance from the subject, which can reduce the viewers' experience quality, and the content coverage on the board if the camera is placed closer. Although, there are many platforms which provide free courses, the technology they use is compromised due to the associated costs.

In some free platform lectures, even though the professors are really enthusiastic there is usually a crew asking them to stay withing the frame :)

Some other applications in which it could prove useful are:

• With increase in remote learning and online workspaces, it is important to maintain clarity of the subject and focus on the required.
• In cooking shows/videos, it's essential to have a close-up view of the chef's actions and also the other parts of the kitchen.
• Technical assembly procedures and health procedures

Ubuntu 22.04 image and Kria KR260 setup

• Download the official SD card image from here which can be found in the Kria KR260 Getting Started Guide which explains the steps to burn the image into the SD card. To explain a bit about this, this is similar to the Windows or Mac in your computer which is put into an SD card as an "image" and the SD card later on is inserted into the board. The board now has the required files to boot into Ubuntu, set up the file systems and all other kinds of good stuff so you can have Ubuntu running on the board. Make sure to enable GNOME desktop as well if you want to use the GUI. If you have a DisplayPort Cable, you can connect it to the board and your monitor(depends on the port of your monitor, preferrably DisplayPort and accordingly get a cable) to have this feature. The rest of the login process and setup is explained in the above link.
• A serial terminal like Putty or TeraTerm can be used. Open up the serial terminal you like and set the mode to Serial in Putty and select the COM Port(you can find this in Device Hardware Manager in Windows) or as TeraTerm likes it, the USB port(like /ttyUSB1 which you can find using lsusb on Linux) after which you will be able to see the login screen after the boot process.
• Make sure to connect to your router with the ethernet to have internet on the board.

Tip: scp <source> <dest_user_name@ip_address:/path_to_destination>-r flag can be used to scp a folder. This command can be used to copy files from your computer to the board provided you have an internet connection.

Vivado flow for PMOD

[Refer to Whitney Knitter's post about Getting Started with the Kria KR260 in Vivado 2022.1 and RPi+PMOD Connector GPIO with Custom PL Design in Kria KR260]

[STEP 1]Launch Vivado and create a project:

~$ source <path_to_Vivado_installation>/Vivado/<Vivado_version>/settings64.sh 
~$ vivado &

source sets the environment variable for that terminal. If you close the terminal, you will have to source Vivado again. But if you close Vivado and you want to open it again you do not need to source it everytime, provided you were working with the same terminal.

• Select the option under Quick Start for Create Project:
• Check the option on the Project Type window that the Project is an extensible Vitis platform, if you plan on making a hardware accelerated design later on.
• In the Default Part page, select Kria KR260 Robotics Starter Kit SOM.
• Click on Connections and for Connector 1 select Robotics Starter Kit Carrier(SOM240_1) and for Connector 2 select Robotics Starter Kit Carrier(SOM240_2) or as per your requirements.
• Click Finish to create the new Vivado project.

[STEP 2] Create a Block Design:

• In the Project Manager window under the IP Integrator tab, select the Create Block Design option and give it the desired name in the following pop up window.
• Click the + button in the Diagram window and add Zynq MPSoC IP
• Add the Clocking Wizard, Processor System Reset, AXI GPIO and AXI Interrupt

[STEP 3] Validate and generate Block Design:

• In the Diagram tab of the block design, click the checkbox icon to run a design validation.
• One critical warning appears about the input interrupt not being connected on the AXI interrupt controller, which can safely be ignored in this case.

Tip: Disable Incremental Synthesis during Synthesis run in Settings under Flow Navigator window

• Select Generate Block Design from the Flow Navigator window and change the Synthesis Options from Out of context per IP to Global.

[STEP 4]Create HDL Wrapper:

• In the Sources window, right-click on the block design file and select the option to Create HDL Wrapper.
• Then select the option to let Vivado manage the wrapper and auto-update it before clicking OK in the pop-up window.

[STEP 5] Generate Bitstream:

• Select to generate a bitstream from the Flow Navigator window
• Click OK to launch the runs for synthesis, implementation, and bitstream generation.

[STEP 6] Export Platform for SW Development:

• Select the option to Export Platform from the Flow Navigator window.
• In the platform packaging windows, select Hardware for Platform Type (the Kria doesn't currently have emulation support) and be sure to check the option to include the bitstream in Platform State.

Moving the motors using PMOD

[STEP 1] Hardware setup:

The pan tilt system can be assembled by following this tutorial and the pan tilt can be purchased here and the servo motor can be purchased here.

Here is a view of how the assembled 2 axis tilt mechanism looks like.

The clamp of the Logitech webcam can be removed by following this tutorial.

Now, we can clamp the webcam to the 2 axis tilt mechanism using rubber bands to avoid using glue :)

This is how the end product looks like

[STEP 2] Rotating the motors using the PYNQ overlay

Transfer the pmod_kria_wrapper.bit and pmod_kria.hwh (found in pmod_kria.gen -> sources_1 -> bd -> hw_handoff ) to the Kria board. Make sure to rename both the .bit and .hwh to a same name to avoid getting unable to parse metadata error

Now that we have the pmod_kria block design and the mechanical 2 axis pan tilt mechanism ready, we can now work on the logic that actuates the servos.

Connect the gnd to the brown wire, 5v to the red wire, and the gpio[2] and gpio[4] to the orange wire of any pmod that is preffered. refer to this diagram for pmod pinout.

Create an Overlay using the pmod_kria.bit and pmod_kria.hwh. Lets create and use the axi gpio instance of pmod1 to access the gpio pins.

Assuming we get the DPU working correctly, on running the YOLOv3 algorithm, we get the bounding box's dimensions (height, width, center_x, center_y) of the detected objects in the input image of dimensions 480*640. Hence we shall come up with an algorithm that keeps a count of the current orientation of the camera and then moves it according to the dimensions of bounding box generated from the camera image, thereby giving us a feedback loop.

Inorder to actuate the servo motors, referring to this link, the servo motors PWM data should be in the frequency of 50Hz (T = 20ms). The physical angle zero of servo maps to Ton=0.2ms (Toff=19.8ms) and the angle 180 degree maps to Ton=2.4ms (Toff=17.6ms).

Here is the algorithm that we came up with that had to be used after the DPU was integrated.

import time
from pynq import Overlay
from pynq.lib import AxiGPIO
from pynq.lib import Pmod_PWM

# Initial angles for vertical and horizontal servos
vertical_angle = 90  # Initial angle for vertical servo
horizontal_angle = 90  # Initial angle for horizontal servo

def drive_servos(vertical_angle, horizontal_angle):
    # Get the GPIO instance
    gpio_instance = overlay.ip_dict['pmod_1']
    gpio = AxiGPIO(gpio_instance).channel1
    
    # Calculate the Ton and Toff for each servo motor
    vertical_ton = 0.2 + (vertical_angle * 2.2 / 180)  # Ton for vertical servo
    vertical_toff = 20 - vertical_ton
    horizontal_ton = 0.2 + (horizontal_angle * 2.2 / 180)  # Ton for horizontal servo
    horizontal_toff = 20 - horizontal_ton
    
    # Write the PWM signals to the GPIO pins for 5 iterations
    for _ in range(5):
        # Control the vertical servo motor
        gpio[4].write(0x1)
        time.sleep(vertical_ton / 1000)
        gpio[4].write(0x0)
        
        # Control the horizontal servo motor
        gpio[2].write(0x1)
        time.sleep(horizontal_ton / 1000)
        gpio[2].write(0x0)
        
        # Make sure the PWM cycle time is 20ms
        time.sleep((20 - vertical_ton - horizontal_ton) / 1000)

def update_angles(center_x, center_y, width, height):
    global vertical_angle, horizontal_angle
    
    # Image dimensions
    image_width = 640
    image_height = 480
    
    # Calculate the center of the image
    image_center_x = image_width / 2
    image_center_y = image_height / 2
    
    # Thresholds for angle adjustments
    horizontal_threshold = 50  # Adjust this threshold as needed
    vertical_threshold = 50  # Adjust this threshold as needed

    # Check horizontal position
    if center_x < image_center_x - horizontal_threshold:
        horizontal_angle -= 5  # Decrease angle to move left
    elif center_x > image_center_x + horizontal_threshold:
        horizontal_angle += 5  # Increase angle to move right

    # Check vertical position
    if center_y < image_center_y - vertical_threshold:
        vertical_angle -= 5  # Decrease angle to move up
    elif center_y > image_center_y + vertical_threshold:
        vertical_angle += 5  # Increase angle to move down

    # Call the servo control function with updated angles
    drive_servos(vertical_angle, horizontal_angle)

Here is a snap of the 2-axis servo tilt mechanism, along with the camera, being driven by the KR260 in action.

Running a face detection model

We chose a Logitech C270 HD USB webcam owing to its ease of integration with Kria KR260 Robotics Starter Kit. The process was as simple as connecting the USB to the board. It can be bought from here

[STEP 1]: Using the PYNQ DPUOverlay:

Installation to PYNQ can be found here along with the pynq-helloworld example and you can access PYNQ Jupyter Notebook by just going to a browser of your choice and then typing <ip_address:9090> and logging in using username as xilinx and password as xilinx.

Tip: Internet connection is not required to access the PYNQ Jupyter Notebook.Just connect the board and your computer with an ethernet cable. Assign a static IP to the ethernet interface of your computer by going to Control Panel > Network and Internet > Network and Sharing Center. Now find the ethernet interface and go to Properties > IPv4 > IP address can be 192.168.2.105 and Subnet Mask of 255.255.255.0. Default Gateway need not be assigned. The same can be done on Linux using doing sudo ipconfig <interface_name> 192.168.2.105. On the board you can give a static IP on the same network range by typing sudo ipconfig <interface_name> 192.168.2.105in the shell. Again, look for the right interface such as eth0 or eth1 on your board side. On your computer's browser now type 192.168.2.102:9090 to open up the PYNQ Jupyter Notebook.

[STEP 2]: Running the yolov3 model on the PYNQ Jupyter Notebook:

The YOLO (You Only Look Once) algorithm is a real-time object detection system that identifies multiple objects in an image with a single network pass. It segments the input image into a grid, predicting bounding boxes and class labels for each cell simultaneously. YOLO is celebrated for its rapid processing and efficiency, making it ideal for real-time tasks. It achieves this by leveraging a unified convolutional network for both object detection and classification.

To deploy YOLO on the Kria board, we use the provided example files and create a DpuOverlay. This requires the yolo.xclbin, yolo.bit, and yolo.hwh files. Since generating a custom build that integrates GPIO access with DpuOverlay wasn't feasible, we used the default gen_platform.tcl file available at and executed the make flow to produce the necessary files.

The YOLO algorithm outputs predicted bounding boxes and their dimensions. We capture frames using OpenCV, process them through YOLOv3 on the DPU, and then utilize the bounding box coordinates and dimensions to control the rotation of servo motors.

Integration Challenges

As discussed in the previous sections, we were able to run the face detection model and turn the motors based on PWM signals. While trying to integrate the two there were some issues faced which we tried resolving to some extent.

[ISSUE 1] DPU Version mismatch in Vivado:FIXED

• Download the latest version of DPU IP from here
• While running Generate Block Design select Global instead of Out-of-context IP

[ISSUE 2] Vitis Platform Vitis_AIE_DIR not found:FIXED

• In Vivado, go to Help > Add Design Tools or Devices and Enable Install Devices for Alveo and edge acceleration platforms and Install Devices for Kria SOMs and Starter Kits

[ISSUE 3] Vitis Platform V++ linker error while trying to generate.xclbin file:

• Export the.xsa from the hardware platform
• Build an application platform and follow the steps from here
• While running the Vector Addition application this issue was faced.

[ISSUE 4] DCP file doesn't exist:

• In this tutorial, while running make BOARD=kr260_som, the DCP file it was looking for in binary_container1 was not found.

• While trying to integrate the DPU using the Vitis Flow, we were facing issues trying to generate the.xclbin as mentioned above.
• While trying to obtain the.xclbin file as in this tutorial, while running make BOARD=kr260_som, we were facing the above mentioned issue.
• While trying to integrate the DPU IP in Vivado Block Design, we followed this tutorial using PetaLinux but couldn't implement the whole system.

Conclusion

Face detection using the yolo-v3 algorithm was implemented by capturing frames from the Video through the webcam and using Vivado block design we were able to add the AXI GPIOs and use it to generate PWM signals in PYNQ overlay and turn the motors. Further, integration of DPU was not possible in the Vitis flow. We can create a PetaLinux image by integrating the DPU in the Vivado block design and the generating boot the PetaLinux image and boot components for the board. Using this, we can integrate the DPU with the AXI GPIOs and using existing Vitis AI Libraries(which can be loaded during while configuring the PetaLinux RootFS) and use it for face detection in the video and also turn the motors using the AXI GPIOs.