AgroWeed Seeker (AWS)

Project motivation

Our overreliance and use of herbicides and pesticides over the last decades as part of the industrialized monoculture agriculture, that without a doubt heavily reduced the worlds food insecurity, lead to evolutionary pressure on weeds and pests to become resistant [1]. In a similar fashion to what is happening in medicine with antibiotic resistant bacteria. The number of herbicide resistant weeds (e.g. pigweed), specifically to the most used glyphosate herbicide, has grown significantly leading to more intensive use of herbicides and lower crop yields causing high economical, and environmental, costs [2]. Emerging alternatives such as biological weed control [3] and preventative physical/mechanical methods [4] are gaining traction. However, they pose challenges such as the need for precise identification of target weeds and labor-intensive processes. Nevertheless, they are very promising solutions and may be the only way forward.

While the potential of advanced technologies in agriculture is widely acknowledged, existing solutions predominantly rely on large and expensive machinery, often coupled with proprietary software. Companies such as Blue River Technology (John Deer), Verdant Robotics and Carbon Robotics have introduced commercial solutions utilizing robotics, computer vision (CV), and machine learning (ML) for weed and pest management. However, these solutions typically entail high costs, complex optical setups, and powerful computing systems, limiting accessibility for many farmers and reducing their autonomy.

This is where this project comes into play. By developing a low cost proof-of-concept robotics platform together with a multispectral imaging system, this project aims to take the first steps into a fully open and accessible solution for modern weed and pest management as well as crop monitoring.

The development and use of a multispectral imaging system for plant differentiation and health monitoring promises much better results over simple RGB (or BW) imaging system. This because plants have a very characteristic spectral reflection signature, especially in the RED to IR band transition, which also changes significantly between healthy and unhealthy plants even before appearing in other bands (before the plant losses its healthy green coloring). This can be seen in the spectral reflectance graph bellow (taken from here, here and here):

[1] https://www.scientificamerican.com/article/weeds-are-winning-the-war-against-herbicide-resistance1/

[2] https://pubmed.ncbi.nlm.nih.gov/18181242/

[3] https://linkinghub.elsevier.com/retrieve/pii/S0261219416300928

[4] https://www.sciencedirect.com/book/9780128098813/non-chemical-weed-control

Project Overview/Outline

This project is divided into a few different parts, together forming the complete proof-of-concept solution for the whole AgroWeed Seeker (AWS) system:

• The sensor platform: A custom developed, low-cost and open multispectral imaging system, using active illumination. This is the system used to acquire and in the future monitor plant health based on their spectral reflectivity.
• The controller/brains: The onboard computer, the brains of the whole system which controls the whole solution, both the sensor acquisition and robot platform. Based on the AMD KR260 robotics platform and connected to a diverse set of sensors and actuators, allowing the save locomotion of the robotics platform and detailed data acquisitions.
• The robotics base: The underlying hardware platform where everything is mounted onto. The base frame is a WLToys 124019 RC car chassis (originally from the MR-Buggy3 kit) with modification that make it more suitable as a robotics platform. It has on top of it a acrylic plate holding all the additional sensors and the KR260.
• The control and application software: The control software is the software stack running on the KR260 to control the motion of the robotics platform as well as determining its inertial frame (position and orientation) as well as its surrounding environment (obstacles and targets) it based on a set of sensors. The application software is the software that controls the application part of the whole system, finding adequate targets to analyze, calculating a path towards it and performing the multispectral acquisition of it.

Over the next sections each of them will be explained in detail, explaining how each of these parts work, how they where developed and how they can be replicated. This is then followed by a section with the performed experiments (and the results) and a section with some demo videos as well as a section with, MANY, plans for future work.

Part 1: Sensors

This section will discuss the sensor platform used and under development for the application of this project, a Multispectral Imaging System. A multispectral imaging system is the acquisition of images in multiple bands in the electromagnetic spectrum, combining elements of spectroscopy with traditional photography. As part of the investigation into this topic, a in-depth article was written and published in this page on my website. A in-depth article on the developed multispectral imaging system shown here will also follow there.

There are different ways of achieving this: using active illumination, different cameras with narrowband optical filters or specialized lenses with multiple narrowband optical filters integrated. The simpler one for a DIY solution is using active illumination, where the acquisition band selection is done through the selective illumination of the subject by a narrowband light source. The image is then captured with a wideband, no optical filter, image sensor. This is the path used as a first experimental implementation for a cheap and opensource multispectral imaging system.

The active illumination system being developed consists of three parts, with the KICAD PCB projects, schematics and housing available openly in the attachments section.

• Multispectral light source: This is implemented with narrowband LEDs, composed of 8 independent light channels each with 6 LEDs in series. This allows the selective illumination in up to 8 different optical bands, depending on the installed (soldered) LEDs.
• Multichannel LED driver: This consists of a baseboard with a MCU that can interface and control up to 8 independent LED drivers. Each of theses LED drivers is on its own little PCB module, allowing different configurations and different drivers to tested. The current version uses 8 equal driver boards, each using the ILD8150 LED driver and configured to drive of up to 750mA at up to 30V. The multichannel LED driver connects to the multispectral light source through a 16-Pin IDE connector and is controlled through a UART interface (the I2C and CAN interfaces are not yet working)
• Imaging System: The imaging system, the camera, is mounted on the backside of the multispectral light source and faces its lens through the center of it. The used camera here is a very cheap USB camera module, using the OV9732 sensor, a low resolution (1MP) RGB camera. This camera is used as a proof of concept and will be exchanged by a more ideal BW Global Shutter camera in the future.

As mentioned, the multispectral light source can be fitted with different LEDs (up to 8 different), allowing the customization of it for different applications. Here the selected bands are the bands of higher interest in plant monitoring, where the reflectivity of plants are higher and change significantly based on their health and other conditions, as well as a white LED used for simple RGB baseline imaging. Bellow is a graph showing the spectral response of the 8 different LED bands used (the IR band is not used yet as those LEDs did not arrive in time for this project):

The physical distribution of these LEDs in the multispectral light source is shown in the picture bellow:

And the full multispectral imaging system, with the camera mounted to it and connected to the driver board:

As can be seen in this picture, the camera connects to the KR260 over USB while the driver board connects to it through a UART connection to the RPI header (UART controller implemented in the PL). The driver board must be powered from a DC power supply with at least 22V, this requirement is imposed by the highest forward voltage drop of the LEDs used in the illuminator, here those are the white LEDs with a Vf= 3.1V, times 6 in series gives ~19 V and adding some margin for the LED driver gives the requirement of a DC in voltage over 22 V (also tested in practice).

Due to how active illumination spectral imaging works, the spectral bands are selected by the illuminating light and not by a optical filter at the camera, ambient light is undesirable as it will modify the captured spectral band. For this reason a imaging enclosure was built for this project, on a crafts weekend with my girlfriend. The enclosure measures about 240x240x240mm, a cube, and is made out of white Foamboard, glued together with hot glue. The front and back side of the cube are open, to allow the box to pass over plants, with dark fabric to block light. The sides also have a fabric skirt to seal it better and make it conform to uneven terrain. Bellow is a short picture montage of the build process:

And the whole box finished up, mounted to the robotics platform, with a sample plant inside is shown bellow:

Calibration and test picture taken with this multispectral imaging system are shown in the results section.

Part 2: Prepare the Brains KR260 (Zynq™ UltraScale+™ MPSoC EV (XCK26))

The brains of this robotics platform is the ZYNQ UltraScale+ MPSoC on a Kria K26 SOMs mounted on the KR260 robotics starter kit development board. This board gives access to a variety of completely programmable GPIOs, through the FPGA fabric of the PL, on 4 PMOD headers and 1 RPI header. In addition if provides 4 USB 3.0 interfaces as well as 4 Ethernet, DP 1.2 and 1 SFP+ port. Besides the incredibly versatile FPGA fabric, the MPSoC contains a quad-core Cortex-A53 APU and 4GB or RAM allowing to run a Ubuntu image or Petalinux. It additional contains a Dual-core Arm Cortex-R5F RPU that can run FreeRTOS and has access to the PL fabric as well. The block diagram of the K26 SOM is shown bellow (source):

This combination of APU+RPU+FPGA on a single SoC/SOM makes it one of the most versatile platforms for developing embedded systems, and extremely interesting for robotics application. Xilinx provides a interesting document highlighting its advantage when compared to similar SBCs with benchmark comparisons for the APU and DPU here. Here this was leveraged by using the PL fabric to implement a DPU and a diverse set of peripherals that allow for direct interaction and control of the RC car that serves as the base of this robotics platform as well as with different sensors and radios. All without requiring an additional board/MCU to make the translation e.g. Pixhawk controller or Arduino, making it more compact and at the same time more versatile due to the FPGA fabric. Need more I2C peripherals? No problem, add another on in the PL. The used sensor uses UART instead? Also no problem, just change it in the PL. All while any of the added peripherals in the PL can be routed to any of its accessible IOs! This makes it an ideal development platform, allowing it to be changed on the fly and adapted to new applications and sensors/actuators without changes to the underlying hardware.

Now, this versatility comes at the cost of a steep learning curve and a more complex setup then with a RPI + Arduino, which was learned over the duration of this competition. The learned lessons and how the brains of this robotics platform was setup and configured will be explained in this section. All the developed software for the KR260 is available on the linked GitHub page in the attachment section. This includes the Vivado project for the custom overlay, the compiled overlay (both with and without the DPU) itself ready to be used/loaded by the KR260, the Python code with test/example codes for different peripherals and interfaces leveraging PYNQ as well as the developed ROS 2 packages.

Custom PL Overlay

As mentioned above, one of the huge benefits and strengths of the KR260 robotics platform is its PL (FPGA fabric) connected to GPIOs (4 PMODs and 1 RPI header). This allows for a completely custom and targeted peripherals layout and selection, choosing only the required peripherals with complete freedom to which pins they are mapped. Quite a dream for any embedded systems developer! To leverage this major feature, a custom overlay must be created for the PL. This overlay can then be loaded in run-time for Linux, adding the desired peripherals.

There are multiple great tutorials on the internet and specifically here on Hackster detailing on how to make a custom overlay for the KR260. Due to this, I'll not go into depth about how to accomplish every little part instead linking to the relevant tutorials and adding my own observations and tweaks. Most tutorials I have found follow the Petalinux flow while I wanted to use the overlay also with the official Ubuntu image, and there the flow changes slightly, mainly by skipping building the Petalinux image.

The basic step-by-step for creating a custom PL overlay that can be loaded through Linux is as follows (based on this post on the Xilinx forum):

• Step 1: Create a hardware project in Vivado, add the correct MPSoC and configure it. Follow this guide to accomplish this guide from Xilinx, with the parts mentioning KV260 replaced by KR260 (the guide works also for more recent versions of Vivado e.g. 2024.1). Or these guides on Hackster step 1 and step 2
• Step 2: Add hooks for future addition of hardware acceleration e.g. the DPU. If the linked Hackster tutorial was followed, some modifications will be required. The clock and reset tree should include 3 domains, one for the PL peripherals and 2 for the hardware acceleration hooks, like shown in the linked Xilinx guide, with a final block diagram shown there. A small modification was made here, all tutorials I followed for the hardware acceleration hooks use clock domains of 200+400 MHz while the DPU can support up to 300+600 MHz clocks increasing performance. I therefore adjusted the clock settings in the block diagram accordingly.
• Step 3: Add the desired peripherals. For this check the first part of this tutorial, up to the Petalinux Flow section (not required for how I used the overlay, in the official Ubuntu with PYNQ). Two important details here, to add more then one interrupt to the AXI Interrupt Controller block, a concat block should be added before it. Second, ALL used Timer peripheral interrupts should be routed the this concat block and not left unconnected as that will cause issues when/if the overlay is to be used for the RPUs (the two Cortex-R5 MCUs in the MPSoC).
• Step 4: Validate the design, add the GPIO constraints files, generate the bitstream and export the platform (with the bitstream included option selected!). This is also detailed in the tutorial linked in Step 3, the steps just before arriving at the Petalinux Flow.
• Step 5: Now here is where it starts to differ a bit, the tutorials now go into configuring and creating the a custom Petalinux image, but this is not necessary if we are only interested in creating a new hardware overlay. For this the Petalinux steps can be skipped and the next step is creating the Device Tree Overlay. A simple guide for only this part can be found here, but most tutorials include it after generating the Petalinux image. Resumed, the steps are: Clone the BSP for the Kria and checkout the relevant version (version MUST match the used Vivado version); Copy the.xsa file generated previously to a new folder somewhere (does not need to be in the Vivado workspace or folder); Open a terminal in this folder and source vivado (e.g. "source /tools/Xilinx/Vivado/2023.2/settings64.sh" or "D:Xilinx\Vitis\2023.2\settings64.bat"); Run "xsct" and write the commands listed in the guide "hsi...."; In the command line, enter the generated folder containing the dts files and run the Device Tree Compiler ("dtc -@ -O dtb -o pl.dtbo pl.dtsi"); DONE!
• Step 6: Create a new folder somewhere and give it the name you want for the overlay e.g. kr260_pixhawk_v3. Copy to this folder the following files: The.bin file from either Vivado or the folder where the device tree was generated in; the compiled device tree file e.g. pl.dtsi; the hardware handoff file from the Vivado project ("<prj>.gen/sources_1/bd/<bd_name>/hw_handoff/<bd_name>.hwh"), this is required if the overlay is to be used with/under PYNQ. Rename all files with the same name, except the file extension part, of the parent folder e.g. kr260_pixhawk_v3.bin, kr260_pixhawk_v3.dtbo and kr260_pixhawk_v3.hwh. Finally, create another file in the folder called "shell.json" with the content shown here.
• Step 7: The above folder is everything required to load the created hardware overlay into the KR260 running Linux (works for Petalinux, Ubuntu and using PYNQ). For this, copy the folder to the KR260, both to a directly where it can be easily accessed from PYNQ and to "/lib/firmware/xilinx".
• Step 8: Use the following command to check if the overlay is correctly created and copied: sudo xmutil listapps. This now should have a new entry with the name of the created overlay. If it does not appear most likely either the json file has an error in it (e.g. punctuation error) or the.bin,.dtbo and.hwh do not have the same name and the name as the parent folder or the folder was not copied successfully to "/lib/firmware/xilinx".
• Step 9: Finally, the newly created and copied overlay can be loaded with the following commands: "sudo xmutil unloadapp" (to unload the old overlay) and "sudo xmutil loadapp <overlay_name>" (to load the new overlay).

To check if the newly added peripherals where correctly loaded and are accessible, the following commands can be used to list them (peripherals from the PL will be listed with a address range starting with 0x8....):

• I2C: i2cdetect -l
• UART: ll /sys/class/tty/ (shows as ttyULn)
• GPIO: ll /sys/class/gpio/
• INT/IRQ: cat /proc/interrupts

This was the flow I used to create my custom overlays, which I successfully used under Ubuntu, Ubuntu + PYNQ and Petalinux. The final custom overlay created added multiple peripherals to the KR260 through the PL fabric, both routed to the 4 PMOD connectors and the RPI header, as well as hooks for hardware acceleration (e.g. for adding the DPU). The distribution, configuration and choice of the used peripherals where to reflect Pixhawk compatible interfaces, one of the most used and open standards for hardware interfaces for drone controllers. In this perpsective, the following Pixhawk compatible interfaces and required peripherals where configured, on the 4 PMOD headers:

• 2 IMU Ports (IMU0/IMU1): Each with independent I2C peripheral with 400 kHz and 2 GPIOs for e.g. interrupt intputs
• GPS/GNSS: Combination of a UART interface with 57600 baud, a I2C interface with 400 kHz, and 4 GPIOs for indicator LED, safety switch and buzzer.
• RC In/PWM Out/MotorControl: A UART interface with 57600 baud to connect to RX receiver over S.Bus; 4 PWM outputs to control servos, motor drivers, etc..; One timer input for motor encoder read-back; 1 GPIO.
• 2 Telemetry Ports (TELEM0/TELEM1): Each with independent UART peripheral with 57600 baud and 2 GPIOs.

On the RPI Header, a I2C peripheral was added to the respective RPI I2C pins as well as a UART interface also on its respective RPI UART pins. All other pins are configured as GPIOs. The picture bellow shows a overview of the mapping of the different peripherals to the headers of the KR260:

In addition, in the attachment section (KR260_Pixhawk_Pin_Mapping.xlsx) there is an Excel detailing the mapping of each peripheral to each GPIO, with the corresponding GPIO name on the FPGA side and schematic side as well as its function and memory mapping. The full view of the Vivado project block diagram is also available there. An empty template file (KR260_Pin_Mapping_Template.xlsx), only with pin naming/mapping between FPGA->Schematic->Hardware, is also available there, to facilitate custom overlays organization.

Adding theDPU (PYNQ-DPU)

Although it is possible to add the DPU in the Vivado flow already, even after multiple attempts following different tutorials I was unable to get both the DPU and my added peripherals working (I will revisit this issues in a later date without the timing constrains of this competition). This prompted the switch to adding the DPU through the PYNQ-DPU flow, which also reinforced the path of using the official Ubuntu image together with PYNQ instead of Petalinux (PYNQ is only easily and officially supported by the Ubuntu image).

The advantage of this flow for adding the DPU is that in Vivado, only our custom design has to be prepared, with the desired peripherals and interconnects, while the DPU is added in at a latter stage. For this, the added hardware acceleration hooks set-up during the Vivado design stage will be used.

With the Vivado platform project built and exported, getting the.xsa file, the DPU can now be added into it through the PYNQ-DPU flow. For this, the guide linked in the GitHub repository can be followed, or this step-by-step guide I ended up with using:

• Step 1: Create a Vitis platform from the created.xsa file. Simply create a new folder with the.xsa file, open Vitis with that folder as the workspace and select create new platform. In the configuration menu, give it a name, select the.xsa file and make the platform for a Linux OS without generating the boot files.
• Step 2: Built the created platform, this will fail with an error related to "no.bif file" but it does not matter in this case. Exit Vitis.
• Step 3: Clone the PYNQ-DPU repository and create a new folder in the "DPU-PYNQ/boards/" folder with the desired name of the new overlay e.g. kr260_pixhawk_v3_dpu
• Step 4: Add the following files to this folder: the.xsa file from Vivado, and "dpu_conf.vh" and "prj_config" which can be obtained from other board files e.g. from "DPU-PYNQ/boards/kr260_som/".
• Step 5: Modify, if desired, DPU configuration file ("dpu_conf.vh") to get the desired DPU specs. I left it as is, configured for the largest DPU size of B4096.
• Step 6: Modify, if required, "prj_config" file to match clock and connections names with the ones given in the Vivado project. Here I had to modify the clock frequency to match my set higher ones in Vivado e.g. "freqHz=300000000:DPUCZDX8G_1.aclk" and "freqHz=600000000:DPUCZDX8G_1.ap_clk_2". The rest already matched the names used in Vivado.
• Step 7: Build the new board, following this guide, from the "DPU-PYNQ/boards/" folder. This is, run: "source /tools/Xilinx/Vitis/2023.2/settings64.sh", "source /opt/xilinx/xrt/setup.sh" and "make BOARD=<Board> VITIS_PLATFORM=<Platform-location>". In the latter, <Board> is KR260 and <Platform-location> is the path to the created Vitis platform.xpfm file.
• Step 8: Get "dpu.bit", "dpu.hwh" and "dpu.xlcbin" from board folder and add it to a new folder, and copy it to the KR260 board. This is now the new overlay folder that can be used under PYNQ to load the DPU and all peripherals. As this overlay is missing the.dtbo file (the device tree file) it can not be loaded trough xmutil and the peripherals will not show up in the Linux device tree. A workaround I used is to load through xmutil the overlay without the DPU, loading the device tree to Linux with all the peripherals and then load the DPU overlay through PYNQ. This keeps the device tree configured in Linux for all the peripherals (except the DPU). Ideally a.dtbo should be created for the DPU overlay so that it can be loaded with xmutil but I have not yet figured out how...

In Step 7 the following error will occur due to the used Vivado version (above 2022.1): "Error: Please source Vitis 2022.1 settings.". To fix this, edit the file "/boards/check_env.sh", modifying where it says "fgrep 2022.1" to "fgrep 2024.1" or whatever Vivado version is installed (e.g. 2023.2).

DepthVision

To add depth vision to the Robotics platform, a cheap IR pattern projection based depth vision module was used. It uses the same hardware as the original Kinect 360 and the ASUS Xtion PRO (but missing the RGB camera). A mode detailed article about this depth vision module can be found on this page on my website.

Getting this depth vision module working on the KR260 with Ubuntu was much simpler then anticipated, only two packages where necessary to be installed, which can be done with the following commands (the second is to allow access to it through Python):

sudo apt install libopenni2-0 libopenni2-dev
pip install openni

With this, the depth vision module is ready to be used. The only error that may be encountered (I encountered it in some instances) when installing the above packages is the following: "/lib/libOpenNI2.so: file does not exist". In those situations, the following tile needs to be edited: "/usr/local/share/pynq-venv/lib/python3.10/site-packages/openni/openni2.py", changing the part under "else: # GNU/Linux, *BSD, etc", change to:

_default_dll_directories += [
			"/lib",
			"/usr/lib",
			"/usr/local/lib",
			"/usr/lib/aarch64-linux-gnu/",
		]

With this I have not encountered any more issues with the use of the depth vision module. The same flow is also usable for the Kinect sensors, which I'm in the processes of acquiring and using as it provides the additional RGB camera. A sample image taken at this step from the depth vision follows bellow:

Installing PYNQs

As already mentioned, due to using the PYNQ-DPU flow and to facilitate development, I choose to use the PYNQ framework. This allows the use of Python to interact with PL peripherals in a simpler way and also with the DPU. The installation of PYNQ on the KR260 running Ubuntu is straight forward, just clone the repository and install it, using the bellow commands:

git clone https://github.com/Xilinx/Kria-PYNQ.git
cd Kria-PYNQ/
sudo bash install.sh -b KR260

After this, PYNQ can be used in Python to load overlays and interact with the peripherals. In case linux based drivers are used in combination with PYNQ, e.g. using pyserial, the overlay should be loaded using xmutil and then in PYNQ the overlay should be loaded without downloading it to the PL e.g. using "ol = Overlay("overlay.bit", download = False)".

As mentioned in the PYNQ-DPU, due to the above reason I load first the overlay without the DPU using xmutil and then in PYNQ I can load, with downloading to the PL, the overlay with the DPU. This allows the use of the DPU in PYNQ while also using other peripherals with Linux drivers (e.g. pyserial).

InstallingROS 2

To control the Robotics platform, and to make it more versatile, I decided to use the very popular ROS 2 framework. Using ROS allows the use of diverse already developed packages to be integrated as well as easily allow expansion of capabilities to the platform. This makes it an ideal choice for a Robotics platform where different capabilities may be added and changed depending on the application and used/added sensors.

Installing ROS on the KR260 running Ubuntu is quite simple as it is the same as installing it on any Ubuntu Linux image. The guide for this can be found here.

After the successful installation following this guide, the only issues that occurred was when trying to use PYNQ in ROS 2, as part of a node. This seems to be an issues of how ROS 2 and PYNQ where installed, with the virtual environment and file paths. The simple solution I found was just editing, through the command line, the Python paths using the bellow command:

export PYTHONPATH='/home/ubuntu/ros2_ws/install/locomotion/lib/python3.10/site-packages:/opt/ros/humble/lib/python3.10/site-packages:/opt/ros/humble/local/lib/python3.10/dist-packages:/usr/local/share/pynq-venv/lib/python3.10/site-packages'

The actual paths may change depending on where PYNQ and ROS was installed. Before editing the path, verify the default used path after sourcing ROS 2 and copy that, adding only the PYNQ path to the end of it. Checking the python path can be done using the following command:

echo $PYTHONPATH

Using the RPU and custom Petalinux

The initial plan for this project was to leverage the RPU cores of the MPSoC to offload the Robotic Platform control loop processing to it e.g. implement a wheel robotics base controller there that takes in velocity commands and controllers the motor and steering servo accordingly (with a PID feedback loop). It would also be responsible for reading the IMU and GPS data as well as wheel encoder data and perform sensor fusion and report back the processed data to the APU. This was abandoned, for now, due to time constrains and multiple issues found that have not yet be solved.

After a long battle, I was able to get a basic FreeRTOS application developed, with some peripherals but no IRQ, and running on the RPU (VMUController). The application is loaded to the RPU from the APU, running custom modified for the purpose Petalinux, with remoteproc. The flow to get this FreeRTOS application is as follows:

• Step 1: Create a Vitis project (File -> New Component -> Create Platform). Set name, Choose Hardware Design and select the previously created XSA. Select Operating system: FreeRTOS; Processor: psu_cortexr5_0. Disable/Deselect Generate Boot artifacts
• Step 2: Create new application component (File -> New Component -> Application). Set name, select created platform and keep domain as R5.
• Step 3: Open Demo projects and driver example projects to get code sniped and create your own application.
• Step 4: Build the application and copy the generated.elf file to the KR260 to "/lib/firmware".
• Step 5: Load the correct overlay in the KR260 (the same as the used.xsa file for the Vitis project, same peripherals).
• Step 6: Run the application with the following commands:

sudo -s
echo VMUController.elf > /sys/class/remoteproc/remoteproc0/firmware
echo start > /sys/class/remoteproc/remoteproc0/state
echo stop > /sys/class/remoteproc/remoteproc0/state

Under the custom Petalinux project (with the modified DTSI file following this) this worked, with the exception of interrupts that posed another challenge not yet resolved, and after connecting ALL timer IRQ pins to the MPSoC (floating ones caused issues building the BSP...). The current issues with the IRQs in the RPU is that when any peripheral IRQ is used/enabled, the interrupt handler would not be called and when using FreeRTOS it seems that ANY interrupt handler stops being called including the Tick Timer... Until now this issue has not been solved and it is one of the reasons for abandoning the use of the RPU for this competition (will pick it up after). Without using the IRQs, the following peripherals were able to be implemented and tested so far, with developed drivers/wrappers for them:

• GPIOs: Read and write but no IRQ.
• Timer PWM: Set timer to PWM mode with set frequency and duty cycle.
• UART: In pooling mode only writing, no reading and no IRQ

Besides these issues, and the solved ones, the next is that I was unable to get remoteproc working with the official Ubuntu image, or get PYNQ working under Petalinux, and the use of PYNQ was essential for the timeline of this project not only because of the DPU but also for faster development with Python. In any case, the basic example FreeRTOS based application for the RPU is also contained in the project GitHub page linked the the attachments section, and its something I'll come back to.

Adventures with creating a custom Linux image

As mentioned in the above section, about running code on the RPU and about the DPU, I run into a conflict between these two scenarios. On one hand I could only get the applications loaded into the RPU with remoteproc using Petalinux and on the other I only could use PYNQ with the custom Ubuntu.

First seemingly simple solution was to install PYNQ on Petalinux, following this guide. But installing it as a packagegroup does not work on new Petalinux (at least 2023.2 and 2024.1 did not work), neither did adding it through the Petalinx build process...

This lead me down the path of trying to merge Petalinux with the Ubuntu image, which others have used to get remoteproc and OpenAMP to work under Ubuntu. But following that guide only leads to a invalid username/password. This issues comes up in the forum as well, with no solution found...

Official Ubuntu Kernel Issue

With the official Ubuntu image, I encountered a issue when updating from the included kernel version (5.15.0-1027-xilinx-zynqmp) to the newest (5.15.0-1031-xilinx-zynqmp). This is an automatic update when connecting the KR260 to the internet and leads to issues when using the custom overlay.

What happens is that there seems to be a clock misconfiguration, as with the new kernel the UART baudrate (and any other peripheral clock) are not matching the programmed ones and differ from when using the original older kernel. I discovered this after many re-installations of the Ubuntu image, with slowly applying updates until it broke the peripherals. I also confirmed this by measuring the output clock from a PWM configured Timer, which with the original kernel gives a programmed 1 kHz signal while with the new kernel this drops to 909 Hz...

The only solution to this I found was to purge the new kernel and downgrade to the original one after automatic updates are applied. The commands for this where found here and are as follows:

sudo apt-get purge linux-image-5.15.0-1031-xilinx-zynqmp
sudo apt-get purge linux-headers-5.15.0-1031-xilinx-zynqmp

To verify and get the installed kernel (image and header), before and after the pruge, the following commands can be used:

dpkg --list | grep linux-image
dpkg --list | grep linux-header

And to check what kernel version is currently used/loaded, the following command can be used (use this after purging the new kernel and rebooting to confirm that the old kernel has been loaded):

uname -r

This solves the kernel problem. I first tried the Xilinx guide of re-building the kernel from source (this could also be used to fix/add remoteproc and OpenAMP I think), but the build process on the KR260 took to long (over 12h and it was not close to finish when I aborted). Cross compilation is another option but the purging the new kernel worked and fixed my issues (also time constrains again makes you choose the fastest path sometimes).

Part 3: Robotics Base

The robotics platform uses as its moving frame a WLToys 124019 RC car. This comes originally from the MR-Buggy3 kit, but with all the original modifications removed and new ones added. On the RC car chassis, the servo and motor was replaced, the servo to a mover powerful one while the motor was exchanged to a geared brushless motor with integrated controller which can be found to purchase here. The integrated controller is powered from 12V and takes PWM signals to set the motor speed and a line to set the rotation direction. It also has a pulse output, which outputs 12 pulses per rotation and can be used for odometry and control loops. The control and feedback signals work with 5V logic level, so a level translator is required to interface them with the KR260, which works with a maximum logic level of 3V3.

The motor is mounted to the WLToys with a custom new motor mount (available in the attachment section) and with a hard to find special pinion gear found here (with 20 Teeth and 6mm hole). Bellow is a picture of the geared motor mounted to the WLToys.

The springs of the WLToys were also replaced with stiffer ones, to support the additional weight to be added to it. The original springs have a wire diameter of 1 mm (and are silver), while the replacement ones have 1.3 mm in the front and 1.2 mm in the back (and are black as can be seen in the picture above). The replacement springs can be found as upgrades part for the LC Racing EMB-1 (which the WLToys124019 is based/copied) e.g. part numbers L6138 (back) and L6139 (front). In general many upgrade parts can be found this way.

To transform the RC car into a robotics platform, a build plate has to be added on-top of it where the on-board computer (here the KR260) and other parts can be mounted to (diverse sensors, radios, cameras...). For simplicity, both to find and to work with, this build plate was made out of an acrylic plate cut to the size of 300x250mm. It is fixed to the RC car with two holders on the shock absorber mounts (back mount can be downloaded from here and front mount from here).

Additionally 6 aluminum M3x60mm standoff are used for increased support. Those fit nicely into the holes in the plastic skirt of the RC car. Bellow is a picture of aligning the acrylic plate and locating the holes to be drilled to screw it to the RC car.

With the holes drilled, the plate is fixed with M3 screws and a plastic washer to the standoffs and shock mounts. Depending on the load-out of the robotics platform additional holes can easily be drilled to mount the desired equipment. Here this will be the following:

• The KR260 development board
• A compact and cheap LiDAR found here (untested)
• A GPS module, the Pixhawk GPS Module (basic version)
• A L-Bracket in the front to mount a camera boom

All these are mounted to the build platform through holes and screws. Bellow is a picture showing them all mounted, without cabling other equipment.

Other equipment can also be fixed by gluing it to the acrylic plate. My preferred option is to use Velcro strips so that it can be removed easily. The equipment mounted this way is:

• Pixhawk Telemetry Radio
• FlySky Mini RC receiver
• Small breadboard

In addition, two plastic 20x20mm L brackets were mounted to the side of the acrylic base. For added protection and to mount the imaging box to it. Bellow is a picture with all the equipment mounted and wired up (mostly):

In the front we can see the camera boom, a 25cm plastic 20x20mm square tube screwed to the L-bracket on the acrylic plate. At the top of it, with the help of a plastic L bracket piece, the depth vision module and the webcam are mounted (hot glued), providing a 3D + RGB vision of the front. To the top (right from car point of view) is the mounted imaging box, with the LED controller on top.

The wiring of all the sensors is as follows (cameras are connected over USB):

• LiDAR -> KR260 RPI UART (Untested and not powered)
• GPS -> KR260 PMOD2 with the GPS UART/I2C/IO
• Telemetry Radio -> KR260 PMOD4 to the TELEM0 UART
• LED Driver -> KR260 PMOD4 to the TELEM1 UART
• RC receiver -> KR260 PMOD3 with the RC UART
• Brushless motor controller -> KR260 PMOD3 PWM output and motor IOs
• Steering servo -> KR260 PMOD3 PWM output

Besides the brushless motor and the LED driver, all are powered from the KR260 either from its 5V supply line on the RPI header or the 3V3 line on the PMODs. The brushless motor is powered from a 12V supply while the LED driver must be powered from a supply over 22V. The idea, which is missing the wiring but the components already arrived, is powering everything from a 5-7s LiPo, with the KR260 and brushless motor powered through a 5A buck converter (this one). This is a task under development at the moment of writing but not in time for the competition. The LiDAR module, currently not used and not powered, will be powered from a dedicated 5V buck converter as its current draw is quite high (around 300-500mA measured).

Part 4: Control and application software

This section explains the developed control and application software. The language of choice was Python for faster prototyping, and using ROS 2, OpenCV and PYNQ frameworks when applicable. All the developed software is available on the project GitHub linked in the attachments.

To run all the developed scripts, including the ROS 2 nodes, the order of operation is as follows:

• Step 1: Load the overlay without the DPU (kr260_pixhawk_v3) using xmutil.
• Step 2: Run/launch a python script that uses the DPU (so the vision perception codes), which will load the overlay with DPU (downloading it to the PL).
• Step 3: Launch all other codes (and ROS 2 nodes) that use PYNQ, and therefore must load the overlay (not without downloading it to the PL). Wait for one to finish before loading the next.
• Step 4: Launch other codes (and ROS 2 nodes) that use Linux drivers e.g. ones that use only pyserial and not PYNQ.

If step 4 is performed before step 3, the serial interface will not be recognized and sometimes can lead to Kernel crashes. Also, when migrating to the latest version of Vivado, 2024.1, the version used for the overlay kr260_pixhawk_v3, after adding in the DPU with the PYNQ-DPU flow, this resulted in errors when trying to use the DPU with PYNQ (DPU not found error although it is in the.hwh file with the correct name). Due to this, for the DPU tests a previous version of the overlay is used, called kr260_pynq where some peripherals differ but the DPU works correctly.

To facilitate development, driver/wrappers were developed for every peripheral available in the custom PL overlay, based on example codes available in the PYNQ documentation here. These are all available in the linked GitHub repository and are the following:

• Driver Camera: This is just a simple helper that reads frames from a camera feed in the background (on a thread) and keeps only the latest. This way when a new frame is requested it is always the latest and not a older buffered one. (Based on this forum post).
• Driver Capture: Functions to interface with a AXI Timer working in capture mode. Intended to be used for the motor encoder. Still has issues with interrupt handling.
• Driver DPU: Functions to interface and use the DPU, specifically for the Yolov3 model. Based on the available Jupiter notebook code here, with some modifications.
• Driver IBus: Functions to read the RC receiver, working in IBus protocol mode, over serial and decode the received messages.
• Driver Illuminator: Functions to interface and control with the multichannel LED driver, over serial.
• Driver PWM: Functions to interface with a AXI Timer working in PWM output mode. Used both to control the brushless motor and the steering servo.

Additionally, quite a few test/example python scripts where developed to interface with different peripherals, for OpenCV use, the depth camera using OpenNI and streaming video using imagezmq.

LocomotionControl

Currently the locomotion controller is quite simple and straight forward. It is implemented as a set of ROS nodes (in the locomotion package). Only the velocity controller and RC remote controller nodes are being used and the whole system is based on a previous project of mine, that can be found here but with quite a few modification to work with the KR260 and the PL overlay peripherals as well as the newly used geared brushless motor. At the time of writing, only the following two ROS nodes are tested and work with this robotics platform using the KR260:

• RC Remote controller node: This node reads the RC receiver messages over a PMOD UART using IBus. Selects the desired channels and scales the value to publish them as a Twist topic (linear velocity and angular rate).
• Velocity controller node: This node listens to a Twist topic and controls the PWM outputs of the KR260 on the PMODs accordingly. Currently this is all done without feedback, no PID loop. Linear velocity commands are simply scaled to a PWM value and output to the brushless motor controller (with the motor direction set correctly). The angular rate is used to compute the necessary steering angle based on the used linear velocity and the Ackerman steering principles. This is then used to control the PWM signal to the steering servo.

As mentioned above, in the velocity controller, due to using a RC car with front wheel steering (like a typical car), the computation of steering angles to the servo motor can be more challenging (also the odometry calculation). This type of steering setup is called Ackerman steering, where the inner and outer steering wheel have a different angle, so that both are tangential with the line connecting the wheel with the steering center. Bellow is a block diagram showing this principle:

The computation of the steering angle to be applied to the servo, based on the twist command angular rate and linear velocity is done with a custom helper class (ackerman.py). More details about this can be found in the linked previous project of mine.

Bellow is a short clip showing the locomotion controller in action. Commands are send from the RC remote, interpreted by the remote controller node and applied by the velocity controller node:

VisualPerception

The visual perception on this robotics platform is achieved with a combination of a depth camera module (Primesense PS1080) and an additional webcam (Logitech C270), both mounted together on a Boom facing to the front of the robot. This combination allows the detection of objects of interest in the RGB feed, using image recognition model running on the DPU, and estimation of its relative position from the robot based on the depth camera information. Bellow is a picture of this Webcam + Depth camera combo:

A webcam had to be used as the RGB camera because the used depth camera module does not have a RGB camera, and therefore one had to be added. This setup is not ideal and will be replaced with a depth sensing module with RGB camera (e.g. Kinect). In the current setup, the RGB and depth feeds not only are misaligned (also occurs with depth+RGB modules), that mused be corrected, but also the RGB camera has a smaller FoV then the depth camera, which must also be corrected.

Custom Python code (VisionPerceptionCalibration.py) was developed to perform the required calibration, both FoV correction and feed offset. The FoV correction is performed by adding padding to the RGB camera until the same FoV as the depth camera is achieved. The offset is controlled by shifting the RGB camera feed, inside the frame with the padding added (left and right padding can differ as can top and bottom). To visually see the alignemnt process, the RGB camera feeds R component is replaced by the inverse depth feed component (smaller depth values give intenser reds). A short video of this process is shown in the results section. Bellow is a picture of the endresult, with the depth information, IR camera (depth camera) and calibrated/aligned RGB camera shown side-by-side.

After the calibration, the RGB camera feed is aligned with the depth feed and therefore each pixel in the RGB feed has an associated estimation of distance from the camera (based on the depth feed). This means that object detected in the RGB camera feed can be given a relative position from the robot. The object detection of the RGB camera feed is achieved with an object detection algorithm running on the DPU of the KR260. At this stage, this object detection algorithm is the YoloV3 model availble from the PYNQ-DPU repository. Based on the Jupiter notebook code, a driver wrapper was created to facilitate the use of the DPU with Yolov3 (DriverDPU.py) was written. A first sample image of running the YoloV3 model is shown bellow, with the bouding boxes drawn and labeled:

Current work at the time of writing is combining the object detection output of the DPU, the bounding boxes, and the depth camera feed to get the distance of the object inside the bounding boxes (VisionPerception.py). The simple way that has been implemented is using the average of the distance values inside the bounding box range as the distance to the object. This of course has a variety of issues and a more robust method is being worked on. A video showing preliminary results of this approach is shown in the results section.

Additionally to a more robust estimation of the distance to target, a size estimation of the detected object is also being worked on. Using the bounding boxes and the depth estimation this is a case of "simple" trigonometry. Finally, a custom trained object detection model will be trained and used, tuned for the application. The initial ground works for this has been laid, the model loaded to the DPU can be changed easily as well as pre- and post-processing can be adjusted for the new model. The custom model will be trained to detect plants and obstacles. The plant detection will be used as the target to drive to and acquire a multispectral image off, while the obstacle detection will be used to avoid obstacles on this path.

Inertial Perception

Currently the only inertial perception that is implemented is GPS based on and is implemented as a ROS node (gnss_receiver.py). The GPS data is acquired from the GPS module over a UART port on the PMOD, the recieved NMEA messages parsed with the use of the pynmeagps python package and then published as a NavSatFix ROS topic. Bellow is a snapshot of one of the captured NavSatFix topics:

Other inertial perception that is being worked on is using an IMU (MPU6050), which already was used to test the I2C interface of the overlay but has not been finished. No data read and published as ROS topics. Additionally the compass module of the GPS receiver will also be incorporated. Finally, odometry data based on steering angles and motor encoder feedback will be added (the code is already there from previous projects) after the motor encoder is incorporated in the locomotion controller.

Spectral Acquisition

The spectral acquisition control software is composed of two parts: the firmware running on the multichannel LED driver board (to be released on GitHub) and the control and managing software running on the KR260 (available in the GitHub project page listed in the attachments). The driver board software is very simple, just taking in UART commands and use those to control with LED is on and with how much current.

On the KR260 side, two simple driver wrapper was made: DriverIlluminator.py which manages the interface to the LED driver board over UART (UART from the PL overlay) and DriverCamera.py which reads new frames from the USB camera module continuously and saves the latest ones so we can get the most recent frame in a simple way. This two drivers are then used for the acquisition (MultispectralAcquisition.py) and calibration (MultispectralCalibration.py) software:

• Multispectral Calibration: This piece of software allows the calibration of the multispectral imaging system, tuning the camera exposure and LED drive currents. This is done by calculating the average value of the pixels in the central area of the image, where a calibration target should be placed, and then adjusting the LED current for each channel until they all give the same average pixel value. This adjustment is done through a keyboard, and calibration results are shown in the Results section.
• Multispectral Acquisition: Here, after programming the camera with the calibrated exposure and LED drive currents, a full multispectral acquisition is performed on command. The system goes through each LED band, turning the LED on and taking a picture. Each of these pictures is then saved as a RGB and BW image (BW using pixel averaging) as well as a mosaic of all these pictures together.

At this moment, only the multispectral images are taken and saved, allowing the creation of a simple plant database over time. In the future this is planned to be used to train a imaging recognition network to identify different plants (crops vs weeds) and the plants health. The use of multispectral images instead of simple RGB or BW, promises a much better plant differentiation and especially plant health analysis. This because, as discussed previously, plants have a very characteristic spectral reflectivity especially in the RED to IR transition, which also changes drastically when a plant is unhealthy.

Experiments and Results

This section contains experiments done for this project as well as obtained results.

Multispectral Acquisition

As already mentioned in the control software section, the multispectral imaging system first needs to be calibrated. For this the camera has its automatic white balance disabled and the exposure set to manual. A calibration target is then placed in the center of the imaging box. This calibration target is made of a piece of foamboard wrapped in 6 layers of PTFE tape (or Teflon tape, used for pipe sealing). PTFE is the material of choice for spectral calibration as it presents an almost ideal flat reflectance over a wide range of wavelength, especially into the IR range. Ideally a calibration target should be used, which has a know and measured reflectance chart, but those are expensive.

With the target in place, the first step is to lower the exposure time until almost a completely dark picture is achieved when no LED is enabled. This is used as a kind of ambient light rejection filter. After that, each LED channel is tuned in turn until they all present the same average pixel value (on the BW converted image) on the calibration target. An example of this for the Far-Red band (band 7) is shown bellow, after removing the IR filter from the camera lense:

As mentioned, the above image was taken in the Far-Red band with the IR filter removed. With the camera out-of-the-box, light in this band is almost completely invisible (filtered out). This led to the discovery of the IR filter that was glued to the lens in the camera, on the sensor side of the lens. The same image as above, with the IR filter still in place, resulted in the picture bellow, which is almost completely dark even with a much higher LED drive current of 100 mA vs 5 mA, showing that the IR filter effectiveness (undesired in this case):

The results of the full calibration, for all spectral bands, is shown bellow, both the to be used BW image and the RGB version. The middle image is a repeated image of band 8 in BW and band 1 in RGB:

Looking at the above picture, it is important to take into account that the white light band, band 1, the target calibration value is 3 times (~225) that of the other bands (~75) as all three RGB pixels are excited in that band and that must be taken into account. Ideally a global shutter BW image sensor should be used, and this is an upgrade that will be done in the future, as well as using a better calibration target (solid piece of PTFE).

After calibrating the multispectral imaging system, test pictures where taken with a variety of plants, from my girlfriends wast plant collection. Bellow is a short video showing the acquisition, with the test subject plant in the imaging box and the multispectral lighting going trhough all bands with the final mosaic picture assembled and shown on the screen (using the KR260):

And bellow is the results of the mosaic for 4 different plant species, in both BW and RGB:

Some interesting patters can already be observed, such as that in the Far-Red band leave patterns disappear, appearing as a uniform "colored" leave. This for now shows that the developed multispectral imaging system is operational and can produce some interesting pictures. The next steps are adding the IR channel (waiting for the LEDS), switching to a BW global shutter camera and creating an open access database of taken images that can then be used to train a image classifier (or other applications). For the latter, our (mine and especially my girlfriends) enthusiasm for plants comes in handy. Both the huge collection of plants we already posses and our tinkering with indoor gardening.

Visual Perception (Targeting)

As mentioned in the software section, the visual perception requires a calibration step to align the RGB camera feed with the depth camera feed. How the calibration software works was already explained there. The calibration step involves moving the RGB camera feed over the depth camera feed (R channel exchanged for the depth feed) until a visually good alignment is obtained. An object, here a watering can and a table leg, are used to have good points of reference for the alignment. A short video showing the depth to RGB camera calibration is shown bellow (screencapture from the KR260):

The obtained calibration values, for offset and padding, are saved and used here one out. The next experiment, using already the calibration values, is the merging of object detection with depth information. As explained in the software section, the RGB camera is used for object detection using the DPU of the KR260 (for now using the YoloV3 model) and the output bounding boxes are then used combined with the depth frame to estimate the position of the detected object in reference to the camera setup. Bellow is a short video showing pre-eliminary tests of this, with the feed of the depth information, IR camera and annotated RGB camera information shown side-by-side while in the terminal the estimated distance to each of the detected objects is output:

The video shows that both bottles are identified correctly by the Yolov3 model running on the DPU, and that after the calibration process, it was possible to estimate there distance from the camera quite well (70cm and 55cm). Due to how this was implemented, a change of the object detection model is quite easy as well as changing the depth and RGB camera setup, facilitating the migratin to a more robust and tuned solution.

Future Work

As mentioned throughout the project, due to time constraints of the competition and the many faced challenges, a lot of work is left to be done. Although the proof-of-concept stage has been reached which was to main objective of this project. Bellow is a list of tasks that are being actively worked on and that are planned for the future:

• Adding IR channel to multispectral imaging system [Should arrive in August]
• Switching the multispectral imaging camera to a BW global shutter [Later this year]
• Start the image database [After switch arriving IR LED, and updated after global shutter camera switch]
• Move to battery power, from a 5-7s LiPo. First will be tested with a Parkside Tool battery (5S), may be to low voltage for multispectral light. [Currently in the works]
• Adding the LiDAR software and power [First tests done, finished by end of August]
• Add motor encoder feedback to control loop -> Switch to PID velocity controller [Next months]
• Add odometry from motor enocder [Next monts]
• Add IMU and incorporate the GPS module compass [Next months]
• Sensor fusion of IMU, GPS, Odometry [End of year]
• Position controller, takes in a target position and uses fused data to drive there [End of year]
• Improved target acqusition from Depth sensor and webcam DPU image classification
• Custom image classification targeted to the application, finding plants to target.
• Change the depth vision sensor to a Kinect with included RGB camera.
• Moving locomotion and inertial sensor acqusition as well as some sensor fusion algorithms of the APU and into the RPU. [In progress]

Updates will be posted over on Twitter and can be followed there, just as was done throughout the competition and will not stop now. This has been an amazing opportunity to start a deep dive into the very interesting, complex and wild world of SoC + FPGA, and custom linux builds.

Acknowledgements

I want to acknowledge and thank the amazing organizers of this competition, especially the Hackster competition managing team, they did an amazimg work keeping everybody in the competition updated and answering relevant questions. I also want to acknowledge the amazin people over in the Discord (especial shoutout to Krishna from LogicTronix) that were always available to discuss ideas and help with issues. I tired to be an active contributing member over there by helping others and contribute with ideas.