An autonomous shieldUAV to protect and save lives

1. Introduction

While the war in Ukraine continues, we notice that low-cost FPV (First Person View) drones and loitering munition (Lancet drones being the most famous ones) have proven especially effective at knocking out multimillion/billion combat vehicles, artillery pieces, tanks, or missile air defense systems.

The FPV drones and loitering munitions are (a) using information supplied by a recon team that finds and locates the targets, (b) based on this information, an operator flies to the spot, (c) confirms the target, and (d) the human operator guides the system to the target and crashes into it. In all this time between these types of UAVs and the base station, where the human operator is placed, video and radio links exist to drive the UAV to see the target and follow the designed path.

For an air-defense system, shooting down a small-sized (two to four kilograms) and high-speed FPV drone (100 km/hour or faster) is not impossible but certainly represents a challenging problem. These drones have a small radar cross-section, and it is not cost-effective to use expensive missiles to shoot down FPV drones (costing around $500) or Lancet drones (costing around $35.000). The cost of different missiles for a specific air-defense system is as follows [1]: (a) IRIS-T - $450, 000, (b) NASAMAIM-9X- $1, 181, 018, (c) Iron Dome, between $20, 000-100, 000, (d) NASAM AMRAAM 120 missile $996, 736, (e) Patriot PAC-3 - $3, 729, 769, (f) VAMPIRE missile -$1, 400, 000. Moreover, cheap drones from various sides can quickly saturate and exhaust an air-defense system.

The most common countermeasures against FPV drones and loitering munition are jammer units placed above important targets or encapsulated as backpack omnidirectional jammer systems for infantry units. These devices jam the command and/or the drone's video link up to the point of losing the communication feeds. As a direct result, the drone operator cannot see (the real-time video transmission) and guide the drone to make immediate decisions and corrections on the fly, especially on the last part of the drone's flight path. However, this approach is no longer feasible.

The latest generation of Lancets is equipped with the NVIDIA Jetson TX2 module, and as a direct result, on the last part of the drone's flight path, an automatic target acquisition and control system is used to hit the target. The jamming systems become almost useless in this mode against partially autonomous drones.

2. Project description

To solve the problem presented above, we developed an autonomous UAV (shieldUAV, Figure 1) that (a) guards a particular area, (b) autonomously identifies other aggressor drones, (c) follows them, and (d) finally crashes into them, thus protecting the objective that was designated as a target to be destroyed by the aggressor FPV drones or loitering munitions. From what we know so far, a similar system does not exist. Since the identification, tracking, and destruction of enemy drones will be done only based on video information, this system is not affected by any jamming technique, and, in addition, a cheap drone will be used to destroy another cheap drone.

Based on this UAV, expensive military hardware will be protected, but much more importantly, human lives will be protected and saved.

This is a very complex project that requires the development implementation of the following hardware and software elements:

1. A gimbal system (composed of a mechanical part, an autonomous electronic and software control system) able to find and track the target - the aggressor FPVdrones and loitering munitions. The need to use such a system will be fully understood when the drone's trajectory control algorithm (based on proportional navigation) will be presented in the following section.

2. Acquisition and annotation of a database with several thousand drone images. This database will be used to train the neural model used in drone detection.

3. Training a YOLO neural model for drone detection.

4. Deploying YOLO on the DPUCZDX8G DPU

5. Implementation and optimization of the UAV video tracking algorithm.

6. Several software components were developed to integrate applications developed in ROS2with the PX4 autopilot running on the FMU (flight management unit).

7. Development, implementation, and testing of the adaptive command and control algorithm of the UAV system.

8. The development, implementation, and testing of all hardware components capable of supporting the operation of all the previously presented elements, all of them placed on the HoverGames UAV system - e.g., power distribution systems (3.6V, 5V, 6V, and 12V), interconnection elements between the various components (USB FTDI connections), input/output blocks, etc.

9. In order to acquire the database and test our UAV system (shieldUAV), a second drone (the one considered the aggressor) was also built.

3. The UAV systems: shieldUAV and aggressor FPV

Goals (of this section & of the entire project):

1. To develop an autonomous UAV (shieldUAV) to autonomously identify an aggressor UAV, track, follow, and crash into them.

2. To develop a secondary UAV (the aggressor FPV) and based it on it to acquire the database used to train the neural model and to be used as a target system in developing the shieldUAV.

The UAVs used in this project have as their base the NXP HoverGames drone, a professional development kit (KIT-HGDRONEK66) that provides all the main mechanical (frame, BLDC motors, propellers, etc.) and electronic (FMU, GPS, telemetry, power distribution board (PDB), ESC, etc.) components needed to build a UAV. This professional developer kit is not entirely functional and includes no software components. The flight management unit (FMU) included in the NXP HoverGames droneis supported by the open-source PX4 autopilot. So, based on this autopilot, the system was developed.

The algorithms to identify and track an attacker drone and command the NXP HoverGames drone were implemented in the companion computer AMD Kria KR260 Robotic Starter Kit. The FMU and the companion computer system were communicated using the RS232 serial protocol, Figure 2. The software components, part of the shieldUAV systems, were developed and sustained by ROS2 Humble. On the AMD Kria KR260 Robotic Starter Kit, theUbuntu 22.04 operating system (OS) sustains all the software components. On the FMU, the PX4 flight stack is on the top of the NuttXOS. To perform its function, the shieldUAVrequires a gimbal-type system capable of identifying and tracking the target UAV. This gimbal-type system, driven by an interface module, is connected to the companion computer through the RS232 serial protocol.

The communication between the shieldUAV and the ground control station were designed and implemented using four communication links.

The first link is based on a telemetry radio transceiver, working on 433 MHz, using the MavLink protocol. This is the main link with the ground control station (GCS). It allows configuring the flight path, supervising and checking the status of the shieldUAV, changing settings and parameters of the FMU without a USB connection, etc. These functions can be done when the UAV is on the ground, but they can also be done dynamically in the air.

The second communication link, a TBS Crossfire 868 MHz long-range link, is a radio controller (RC) type link through which a ground control operator can take any time control of the drone, being able to switch from the autonomous mode to the manual mode or vice versa.

The third link is used only to evaluate and debug the developed application. Based on this link, we can stream frames from the companion computer, in real-time and in-flight, to the ground control center. To do this, we used the ZMQ or ZeroMQ messaging protocol; this protocol has the following two advantages of ZMQ: it focuses on high data throughput and low latency. This link is based on Wi-Fi technology and is carried out on the UAV through an ALFA AWUS036ACH module. Since the Ubuntu operating system does not support this Wi-Fi module, the drivers were installed by following the steps presented here: https://github.com/aircrack-ng/rtl8812au/.

The last control link, the fourth, consists of a First-Person View (FPV) system that is a method through which, based on information provided by the onboard camera, a UAV can be controlled from the pilot's viewpoint, Figure 2. In this project, the FPV system was mainly used to pilot the shieldUAV manually and acquire the image database.

An intuitive video presentation of the shieldUAV is the following one:

We also developed a second drone, the aggresor one. This drone will "attack" the positions guarded by the shieldUAV. It was used to create the image database used to train the neural algorithm and test the shieldUAV drone. A presentation video of both drones (the attacking one and the shieldDrone) is as follows:

Encountered problems:

The Kria KR260 Robotics Starter Kit has many electronic components on the back side of the PCB board. So, it is very easy to destroy them. We actually managed to do this.

In order to protect these components and to be able to fix Kria KR260 on a carbon fiber surface (a conductive one), we made the following 3D printed element.

This design was openly published on the Thingiversesite:

https://www.thingiverse.com/thing:6699563

Obtained results:

As both previous videos have presented, both UAV systems have been developed, tested, and are functional. In the rest of this report, we will show the implementation of the different components and the results obtained.

4. Proportional navigation guidance system

For a missile or for a UAV (like in our case) to be able to intercept a target, they must navigate toward a point in space ahead of the target system and must arrive there at the same time moment when the target arrives.

Proportional navigation guidance law is based on an observation made by sailors. The sailors observed that if (a) two ships are moving, and (b) from an observation point placed on the first ship, the second ship appears stationary, and (c) the size of the second ship is growing, then the two ships are on a collision trajectory.

To intercept the attacker drone – the target (red), the defender drone (blue) must keep its dynamic in such a manner that all line-of-sight (LOS) angles must be equal: alfa(t1) = alfa(t2) = alfa(t3) = alfa(t4). These equalities are the mathematical relations that express the (b) and (c) requirements presented above. The line-of-sight angle is the angle between the line-of-site (defined as the imaginary line connecting the pursuer and the target) and the trajectory vector of the defender UAV.

To implement the targeting system, based on a proportional navigation guidance law, a 3D gimbal is required to point a camera directly to the target. Defender drone uses the pan (the horizontal) and tilt (the vertical) camera angles related to its body frame to control its flying dynamics.

5. Developing 3D gimbal system and the I/O interface board

Goals:

1. To develop a 3D gimbal system (based on two pan/tilt servos) able to drive a camera to any point in front of the shieldUAV system;

2. To develop an I/O interface system capable of controlling the 3D gimbal (through the 2 PWM signals required for the servo motors), providing different visual information on 3 LEDs, and getting user commands based on two push buttons.

A gimbal system is a main requirement for any system guided by a proportional navigation law. The final goal of the gimbal is to track an object (the target) and to pan and tilt the camera in a specific way so that the target UAV will always be in the center of the image. This section of the report will present the gimbal system's mechanic, electronic and software components. In the next section ("The object tracking software component"), the software components able to acquire images, detect an object, and supply the commands for the gimbal system will be presented in detail.

The gimbal system's tracking part is composed of two servos (SR431 type) and a mechanical system. In our case, we have one servo for tilting up and down and a second one for panning left and right.

Each of our servos has a range of 90 degrees, unobstructed by the mechanical part (the fixture) used to keep them together, fix them on the UAV system, and secure the camera. We used only the bottom part of a Tinkerkit Braccio robot arm for the gimbal's mechanical part and removed sections S1, S2, S3, and the gripper (see the following figure left part). Section S1 was replaced with a 3D-printed part we designed (the right part is in Figure 7).

We tried several solutions to control the servo motors. In the first approach, we used the PMOD port of the Kria KR260 board. Unfortunately, we were unable to integrate this solution within the entire project.

In the end, the solution we chose was the development of a new input/output (I/O) interface system (supported by MSP430G2553 microcontroller) capable of controlling a 3D gimbal (through the PWM signals supplied to the stepper motors), providing different visual information related to the UAV state based on 3 LEDs and having two inputs through 2 push buttons. The I/O interface system is connected to the control system (Kria KR260) based on a serial link tunneled through a USB connection. The schematic of this I/O interface system is presented in Figure 8.

Several applications were developed to drive the I/O interface system.

The first application was developed in the LabWindow CVI environment, see Figure 9. The serial port can be configured based on the "Configure" button. A new panel is loaded by pressing this button, and now the baud rate, parity, number of data bits, number of stop bits, etc., can be configured for a specific port. On the main interface, a user can select the PWM duty cycle (between 5% to 10% at 50 Hz), the state of a specific LED (on, off, and blinking), and visualize the push buttons state through the "Read data" interface. The code of this application can be downloaded from https://github.com/dmdobrea/shieldUAV/tree/main/02_LabWindowsCVI_code.

In the final step, a Python application was made to ease the communication with the microcontroller by providing functions for setting the desired state of the servos and LEDs and then sending a message to the microcontroller with the settings. This application is also capable to get the state of the buttons placed on the I/O interface system, Figure 8. The code of this application can be downloaded from https://github.com/dmdobrea/shieldUAV/tree/main/03_IO_system.

The functions LED_state_set and PWM_value_set set the values of the variables with the desired state which is being sent to the microcontroller by calling the update_state function. The call of the update_state can be skipped if at the creation of the object automatic_update was set to True. This guarantees that update_state is called automatically each time something has been set, although it may be turned off if control is wished over when the message for the microcontroller is sent.

The sending of the message is done though the "serial" python library. By creating a serial.Serial() object you can send bytearray variables through serial.Serial.write(to_send).

The reading of the buttons is done at the time of the call of the read_buttons function. It reads from the buffer of the serial port all the data it received by using serial.Serial.read() and it returns how many times each button has been pressed since the last time it was called.

For this system to be functional, we also developed a program (based on the Code Composer Studio environment) that runs inside the MSP430G2553 microcontroller. The code can be downloaded from this link: https://github.com/dmdobrea/shieldUAV/tree/main/01_uC_MSP430G2553_code. This code runs entirely based on interrupts. The microcontroller is waiting for a byte to be received. Whenever a new byte is received, an interruption is generated, and within this interrupt function, the sequence of received bytes is decoded. To change the duty factor of the PWM signals and the state of the LEDs, the microcontroller accepts the following serial sequence:

The PWM fields can take values between [1000, 2000] – 1000, represent 1 ms, and 2000 is 2 ms. In the LED field, each LED is represented by two bites (0 is off, 1 is on, and 2 is blink). Each time when one of the two push buttons is pressed, an interrupt is generated, and the microcontroller sends the following sequence of values:

In the LED field, each push button is represented on 4 bits. If the button is pressed, the 0x0F value is sent. In the other case, a zero value will be sent.

All the steps taken to obtain the final version of the 3D gimbal and the I/O interface board are presented synthetically in the following video:

Encountered problems:

1. For a UAV (quadcopter in our case) to fly forward, we need a forward component of thrust from the rotors. So, the rotation rate of rotors in the rear is higher than that of rotors in the front. In this mode, the drone tilts to the front; see Figure 12.

Based on the first implementation of the 3D element, the video camera was oriented only downwards during the flight, even if the tilt angle was in the higher upper position, see Figure 12. To solve this issue, the 3D element was redesigned, being longer and having a tilted face surface where the video camera is placed, Figure 13.

2. In the first approach, we wanted to use the PMOD port (BaseOverlay('base.bit')) for the PWM signals and, at the same time, to load the neuronal network on the DPU (DpuOverlay('dpu.bit')). After going through several tutorials and getting some small successes, we didn't find a way to create a functional Vitis design that includes GPIO and DPUIP. This fact led us to develop our own I/O interface board.

3. A very interesting issue appeared in servo motor control, one issue that generates movement in steps with small jerks. Moreover, due to this issue, a sweep of the entire movement space takes around 30 seconds - in a normal case, the same movement takes 1-2 seconds to complete. The problem and the solution are presented in the following video:

Obtained results:

A fully functional 3D gimbal system was developed and tested. Also, an input-output interface circuit was developed, tested, and integrated into the UAV system.

6. The object tracking software component

Goal:

This section presents the development of an object-tracking system that can follow a moving object (a bottle) in a scene while keeping the object centered in the camera's view. The detection and classification system is based on a YOLO deep neural network executed on a DPUCZDX8G Deep Learning Processor Unit placed inside Kria KR260.

The object tracking software component is required to find the LOS (line-of-sight) to the target drone, see Figure 5. Based on LOS, two angles were determined (Figure 14): the azimuthal and polar angles. To obtain a collision trajectory, the shieldUAVdrone must generate flight dynamics of such a nature that these angles must be constant.

The code for the entire application can be downloaded and analyzed using the following link: https://github.com/dmdobrea/shieldUAV/tree/main/04_object_tracking. The next video shows the operation of the tracking system and its components:

The tracking system is based on two PID (Proportional Integral Derivative) control loops designed to reduce the positional error between the center of the acquired image and the center of the detected object. So, the goal is to move the servos in a specific way to keep the detected object centered in the frame.

The project is composed of four Python files:

(A)object_center.py, based on a YOLO model, detects and calculates the center of the bounding box surrounding the detected object – a bottle in our case. The tf_yolo3_voc.xmodel model from PYNQ DPUexamples placed inside the Jupyter notebooks was used. PYNQ DPU was installed accordingly, as instructed in https://github.com/amd/Kria-RoboticsAI. All were supported by Vitis-AI 3.5. The use of this class is very straightforward. First, the ObjCenter must be instantiated:

obj = ObjCenter("tf_yolov3_voc.xmodel")

All parameters are initialized – anchor list, classes, neuronal model is loaded into DPU, inputs, outputs, etc. Each time when a new detection is required, the update method must be called:

((objX.value, objY.value), rect, alg) = obj.update(frame, (centerX.value, centerY.value), 4)

This method returns object coordinates and the bounding box assigned to the detected object.

(B)pid.py keeps the PID class. The main function of this class is the update() method, which returns the value used to drive a servomotor based on the error (existing between the center on the acquired image and the center of the detected object), and on the proportional, integral and derivative terms computed by using the proportional, integral and derivative gains.

(C)mctrl_comm.py contains inside the class presented in "Developing 3D gimbal system and the I/O interface board" section. Based on PWM_value_set() method, the duty factor of the PWM signals sent to servos through the I/O interface system can be easily set.

(D) The last Python's file, object_tracking.py, contains the main program. The main program creates four additional processes: (a)two different processes that incorporate panning and tilting PID control loops, (b) one process used to find and classify objects based on YOLOv3 deep neural network, and (c) the last process, is used to drive the servomotors based on the information obtained from the pan and tilt PID processes. In the obj_center function, an image is first acquired. Second,the bottle is detected in the frame. Third, panning and tilting angles are calculated; the servomotors are driven accordingly.

Encountered problem:

The major problem of the developed tracking system is that it is based on a weak detection algorithm - even if it is a deep neural network one. The practical example presented previously shows that the detection of the bottle by the YOLO algorithm (Attention! Only the blue detections - we will talk about the red detection in the "UAV video tracking algorithm" section of this report) was lost in several frames. Furthermore, the model was trained only with bottles in a vertical position. These facts led us to find an alternative method of improving this algorithm - the technique will be presented in the "UAV video tracking algorithm" section.

Obtained results:

1. We developedan efficient object tracking system that is able to follow a moving object (a bottle) in a scene while keeping the object centered in the camera's view.

2. Moreover, the software components of this system were developed as some independent modules (independent classes) - e.g., the DNN type neural network was encapsulated in the ObjCenter class, while the PID control algorithm of the two servo motors was encapsulated in the PID class.

3. We also determined the unique values ​​of the two PID adjustment loops. These values are specific to the gimbal-type mechanical system and its loading with a particular video camera.

7. Database acquisition and annotation

Goal:

The main goal was to acquire a large enough database (minimum 1500 images for each class as it is recommended) with the help of which we can obtain high-quality YOLO neural models.

A pretty important part of our project was database collection and annotation. We need a way to get images using the camera mounted on the drone and then to annotate the captured images.

For image acquisition and saving we used the OpenCv Python library. We made this choice because it has features relating to getting frames, image processing and image saving, thus simplifying the implementation of needed programs. As a beginning we made a program that only took images from the camera, displayed it and then saved it.

The first challenge was how to control such a program without being able to access the user interface (UI), since the filming will be done on a field without any monitors or auxiliary keyboards and mice. A first method is using a SSH connection to the development board and starting/stopping the program in that way. That might have been good enough, had we needed to only start and stop the program once, but, unfortunately, each time we change the battery of the drone, the development board restarts, because it is powered by it.

The method we ended up using was based on the PMOD’s functionalities. The thing that it proved to be useful for was reading the state of two buttons and being able to give feedback to the user by the actuation of two LEDs, see Figure 15. One button was used to start image acquisition from the ethernet video camera, and the second one to start the same process but this time from the USB video camera. The LEDs were used to present the status of the image recording program (recording, not recording, or an error has occurred) and source if image acquisition (Ethernet or USB video camera). We used the accompanying library for the development board (KV260) to set up the FPGAto communicate with the PMOD and the PYNQ library to map the pins of the PMOD to some variables that can be called to check their status or to set their status (in case of the LEDs).

The image acquisition program can be downloaded from here: https://github.com/dmdobrea/shieldUAV/tree/main/05_img_ACQ.

Now that we have a way to interact with the user, we need to have a loop that checks the buttons and sets the LEDs, but aside from this we also need the loop for frame acquisition and saving. Since the saving process may be slower, it can slow down the responsiveness of the input and output methods if put together in a loop, therefore we decided to separate the two in different processes, which lets Python separate them on different cores of the processor.

With that out of our way we started testing and realized that there existed a very noticeable delay from the ethernet camera. It seemed to have a queue built in from which gave frames. The workaround we found was having a separate thread on the frame capture process that continuously consumed the frames, the frame_eaterfunction. In order for the two threads to not get the frame at the same time and have a conflict with each other we used a mutex shared between the two that locks the other from being able to also get a frame and unlocks it when the frame was acquired.

At this point it seemed perfect, but a small issue still existed: sometimes, stemming from forgetfulness, we took out the battery of the drone before pressing the button for the filming to and irreversibly corrupted the captured footage. The solution for this problem was simple: in the folder of each separate filming, we split the clip into smaller chunks of 20 seconds, so that in the case of a forgetful user or a crash, that ejected the battery, only the last few seconds would be lost. The way we did this was by using a thread on the faster looping i/o checking process that at each 20 seconds would increment a variable that was shared between it and the filming thread. When the filming thread saw the variable change value, it knew to destroy the current writer object and create a new one with a different name of the file in which it was writing (e.g., ETH-3.avi into ETH-4.avi).

The last thing to do is make it start when the development board boots up, because otherwise we’d have to start up the program using SSH, defeating the whole purpose of using buttons and LEDs. We do this by making a service that at the Linux boot runs a shell script that starts the program.

Now, with the program up and running, it’s time for the fun part: going after other drones and planes to capture images from all possible angles. For this we used another drone just like the one we use as defender, more specifically a NXP Hovergames drone, a plane model similar to a Cessna and a glider – see Figure 16.

Unfortunately, for the two planes we couldn’t use the NXP Hovergames drone due to it being too slow to maneuver and safely dodge a possible collision. That’s why we used the GEPRC CineLog35 drone for it’s nimbleness and speed. It couldn’t carry the development board and the camera, so we used a GoPro Hero 11 Black for the capture of the images.

As a direct result we acquired a number of 5328 images. From these, 2481 images are with drone (46.57% from the database), 1340 images with planes (25.15% from the entire database) and 1507 images were negative samples (images that don't contain planes or UAVs), a share of 28.28% from the database.

The entire database occupies 2.9 GB. And will be readily available for as long as possible on Google Drive and can be accessed https://drive.google.com/file/d/1r81qBCOaqhQVSDUNyJjoLjQEiGdBrhz4/view?usp=drive_link.

With the images collected, we used Darkmark for the annotation of the images, since it is already integrated with darknet by the fact that it creates automatically the needed files for training, such as the.cfg file and the paths to the training and validation sets. It also tiles and resizes the images that are too big, so that there is no need to do dynamic resizing during the training, which speeds up the process considerably. Apart from that it creates cropped and zoomed in or out images, which should strengthen the neural network.

Encountered problems:

1. Due to the efforts made in the development of the shieldUAV, we no longer had the necessary resources (especially time, but money also) to build an airplane with the help of which we could obtain a suitable number of images for this class as well. For this reason, the two classes are not balanced, and the aircraft class has the fewest images. In this way, we would like to thank the colleagues from the Iasi model aircraft club who helped us in the acquisition of airplane images.

2. When the service (created by us) at the boot time launched our image acquisition application, we got a series of errors due to the inability of our application to find the various used libraries. To solve this problem, we were forced to run our program with the Python installed in the following directory: /usr/local/share/pynq-venv/bin/python3.So, the scrip was as follows:

#!/bin/bash

cd /home/ubuntu/kamikazeUAV/teste/serviciu
source /etc/profile.d/pynq_venv.sh

/usr/local/share/pynq-venv/bin/python3 imgAcq.py

Obtained results:

Following the activities carried out in this section of the project, we obtained an annotated database with 5328 images of airplanes (1340 images) and UAVs (2481 images).

8. YOLO model training and evaluation

After annotating all the images needed, we started training the neural networks. We first had to train on a personal computer using a GeForce RTX 2070 SUPER™ WINDFORCE OC 3X 8GB VRAM. But, when we had relatively few images, the personal computer was fine, but as the images accumulated, we started seeing slowdowns. Soon, the low amount of VRAM of the graphics card began to become a problem: we needed to lower the batch and subdivisionssettings in the config file of the training of the neural network, and in the case of the YOLOv4 model, it refused to run even at the lowest batchand subdivisions values. This is a very interesting aspect if we take into account the fact that all other neural models (YOLOv3-tiny, YOLOv3, YOLOv4-tiny, YOLOv7-tiny, and YOLOv7) could be trained without problems on the GeForce RTX 2070 graphics card.

While training, we had to tweak the training variables such as momentum, decay,learning_rate, burn_in, max_batches, steps, and scales in order to maximize the performance of the resulting neural network. By tweaking these variables, we tried to get from a graph that looks like the one on the left to one on the right, Figure 17. In short, we wanted more precision from the neural network in the validation set (red line – higher being better). The blue line is the loss from the neural network in the training set.

As the data set grows, training neural models takes more and more time - especially huge neural models like YOLOv3, YOLOv4, or YOLOv7. This led us to find alternative solutions. In the end, we got access to a more powerful machine with four NVIDIA Tesla A100 Ampere 40 GB Graphics Processor Accelerator - PCIe 4.0 x16 - Dual Slot that considerably sped up the training process.

The learning curves in Figure 17 were obtained in the initial training when the database did not have the expansion of the one at this moment (5328 images), and we only have images with one class – UAV.

In what follows, I present the learning cubes for training the YOLOv3-tiny neural model on the complete data set. On the left, we have the results for the UAV class, and the results for the plane class are presented in the right part. The imbalance of data and the insufficiency of the training images for the pane class can also be seen in the much lower performances obtained for this class – right graphic.

The global learning performance for both classes is presented in the following figure.

I close this presentation by displaying the numerical values ​​that characterize a learning process for the two classes:

We obtained similar values ​​and graphs for the other neural models (YOLOv3, YOLOv4-tiny, YOLOv3, YOLOv7-tiny,and YOLOv7). Still, since the only neural model that can meet our application's requirements is YOLOv3-tiny (as we will show later in this report), it is the only one whose results we have presented in detail in this section of the report.

9. Deploying YOLO on the DPUCZDX8G DPU

Goal:

To getyolo.xmodel file (which we needed to deploy the YOLO model on the DPU) from the yolo.cfg and yolo.weights files obtained as training in the DarkNet framework.

The DPUCZDX8G is the Deep Learning Processor Unit (DPU) designed for the Zynq® UltraScale+™ MPSoC architecture supporting the Kria KR260 Robotics Starter Kit. Even if DPUCZDX8G is a configurable computation engine optimized for convolutional neural networks, the YOLO models obtained previously (see section "YOLO model training and evaluation") cannot run directly on the DPU. So, a conversion process is required.

So, the main goal of this subchapter is to present all the steps we did in order to obtain the yolo.xmodel(the file that we needed to deploy the yolo model on the DPU) from the yolo.cfg and yolo.weights for YOLO versions 3, 4, and 7 – obtained based on the Darknet framework. Let's start:

1. The starting point is based on yolov{x}_best.weights(the best weights obtained after the training of deep neuronal model) and yolov{x}_train.cfg (deep neural network architecture description) files. In our case x = 3, 4 and 7 for YOLOv3, YOLOv4, and YOLOv7 neural models.

2. The conversion to keras.h5 model requires using the following repository: david8862 Keras Model Set. This repository can be found here: https://github.com/david8862/keras-YOLOv3-model-set. So, clone it:

$ git clone https://github.com/david8862/keras-YOLOv3-model-set

3. Use the following to create the keras.h5 model (in our case, the working folder with the neural models was placed in the same folder where keras-YOLOv3-model-set was installed):

python3 ../keras-YOLOv3-model-set/tools/model_converter/convert.py yolov{x}_train.cfg yolov{x}_best.weights yolov{x}_UAV.h5

As a direct result, two files have been generated (for YOLOv3and YOLOv4): (1) the Keras model in H5 data format yolov{x}_UAV.h5 and (2)yolov{x}_UAV_anchors.txt file.

For YOLOv7, an error was generated:

ValueError: Unknown activation function `swish` in section convolutional_0

Looking for information about this error, we found that "Swish" activation is not provided by default in Kerasenvironment. As a direct result, we replaced all "Swish" activation layers in the cfg file with "LeakyReLU" (activation=leaky). Of course, using the "Swish" activation function produces the best results, with the highest mAP and lowest validation losses. Still, the only way we discovered to get a model compatible with the DPU from the Kria KR260 is to use a "LeakyReLU" nonlinearity.

Now, having the HDF5 File with extension .h5, the deployment on the DPU generally follows the following steps:

A. Based on the Model Inspector, check if the original model is compatible with the target board's AMD Deep Processor Unit (DPU) architecture. This step is optional but will help you understand if it is required to modify your DNN structure. You can download this tool from https://github.com/dmdobrea/shieldUAV/blob/main/06_YOLO_conversion_to_DPU/inspection_Model.py.

B. Run the Model Quantization process to generate an 8-bit fixed point model ("int8" model) from the original 32-bit floating point DNN. You can download this tool from https://github.com/dmdobrea/shieldUAV/blob/main/06_YOLO_conversion_to_DPU/quantizing_Model.py.

C. Run the Model Compilation process on the int8 model to generate the .xmodelmicrocode for the DPU placed on the Kria KR260 target board. You can download this tool from https://github.com/dmdobrea/shieldUAV/blob/main/06_YOLO_conversion_to_DPU/compiling_Model.sh.

Essential: all these steps, except the last one from sub-point D, must be done on a personal computer or laptop, NOT ON THE Kria KR260 development BOARD!!!! Now, let's continue with the remaining steps:

4. We cloned the Vitis AI repository to the host machine (PC or laptop) as follows:

$ git clone https://github.com/Xilinx/Vitis-AI
$ cd Vitis-AI

5. Then installed the Docker:

$ sudo apt install docker.io

The Docker daemon always runs as the root user. Non-root users must be added to the docker group. We recommend doing the following it at this point to avoid issues:

$ sudo usermod -G docker -a mdobrea

6. Next, to be sure that all is OK, a quick and simple test of your Docker installation by executing the following command is recommended:

$ docker run hello-world

By running this command, the Docker will download and run a test image from Docker Hub in a container. Ultimately, when the container runs successfully, it prints a "Hello World" message and exits.

Finally, we verified the version of Docker that is installed:

$ docker --version

7. Now, we downloaded and installed the most up-to-date version of the pre-built Docker with TensorFlow 2:

$ Docker pull xilinx/vitis-ai-tensorflow2-cpu:latest

8. Then, to start the Vitis AI Docker, we used the following commands:

$ cd <Vitis-AI install path>/Vitis-AI
$ ./docker_run.sh xilinx/vitis-ai-tensorflow2-cpu:latest
$ conda activate vitis-ai-tensorflow2

9. If you want to get out:

$ conda deactivate
$ exit

At this moment, all the required components of the environment are set and functional. We only needed to copy the Kerasmodel in H5 data format, the Model Inspector tool, the Model Quantization tool, and the Model Compilation tool inside the environment.

10. The cp command can be used to copy files from/to the container. A file can be copied to the container like this:

$ docker cp file.txt container_id:/file.txt

A file can be copied from the container like this:

$ docker cp container_id:/file.txt file.txt

In our case, container_id is a container ID, not an image ID. So, we used "docker ps" to view a listing that includes container_ids. Customizing for our specific situation:

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ac48b09f5ffb xilinx/vitis-ai-tensorflow2-cpu:latest "bash" 3 minutes ago Up 3 minutes kind_margulis

$ docker cp yolov4.h5 ac48b09f5ffb:/workspace/my_model/build
Successfully copied 259MB to ac48b09f5ffb:/workspace/my_model/build

11. Next, we will inspect the model. Model Inspection indicates why the layers are not mapped to the DPU. To achieve this objective, we created a special Python especially made for this:

$ python3 inspection_Model.py --keras_model build/yolov4_UAV.h5

The inspection_Model.py can be downloaded from here. Now in inspection_Model.pyfile please modify the following line:

input_shape=[1, 416, 416, 3]

inserting your specific image dimensions.

If all the layers are supported and they can be successfully mapped to the DPU, you should see something similar to these last lines:

For YOLOv3 and YOLOv7, we get something similar:

======================================================================================
[SUMMARY INFO]:
- [Target Name]: DPUCZDX8G_ISA1_B4096
- [Total Layers]: 252
- [Layer Types]: InputLayer(1) Conv2D<linear>(75) BatchNormalization(72) LeakyReLU(72) ZeroPadding2D(5) Add(23) UpSampling2D(2) Concatenate(2)
- [Partition Results]: INPUT(1) DPU(251)
All layers are supported and successfully mapped to DPU.
======================================================================================
[NOTES INFO]:
All layers are supported and successfully mapped to DPU.
======================================================================================
[VAI INFO] Start plotting model to /workspace/my_model/log/model.svg
[VAI INFO] Inspected model has been plotted in: /workspace/my_model/log/model.svg.
[VAI INFO] Start dumping inspected results to /workspace/my_model/log/inspect_results.txt
[VAI INFO] Inspected results has been dumped in: /workspace/my_model/log/inspect_results.txt.
[VAI INFO] Inspect Finished.

For YOLOv4 (tiny version):

Exception encountered: Unknown activation function: 'mish'.

To solve this problem, the solution is straightforward: go to the yolov{x}_train.cfg file and replace "mish" string with "leaky".

But, even if the previous error was solved, another problem appears – in the inspection logs we get:

[SUMMARY INFO]:
- [Target Name]: DPUCZDX8G_ISA1_B4096
- [Total Layers]: 76
- [Layer Types]: InputLayer(1) ZeroPadding2D(2) Conv2D<linear>(21) BatchNormalization(19) LeakyReLU(19) Lambda(3)
Concatenate(7) MaxPooling2D(3) UpSampling2D(1)
- [Partition Results]: INPUT(1) DPU(72) CPU(3)

So, three layers are not supported by DPU and will be mapped to run on CPU. These layers are "Lambda" layers. We were not able to find a solution to this problem until we wrote this report.

12. For the next step, we installed 'imutils':

$ pip3 install imutils

This step performs the float model's post-training quantization (PTQ), including model optimization, weights quantization, and activation post-training quantization. To achieve this step, we developed a new Python program: quantizing_Model.py. The quantizing_Model.py can be downloaded from https://github.com/dmdobrea/shieldUAV/blob/main/06_YOLO_conversion_to_DPU/quantizing_Model.py. So, let's start the Model Quantization:

$ python3 quantizing_Model.py --dataset ./calibrare --keras_model build/yolov4_UAV.h5 --quant_file build/yolo4_UAV_q.h5

Very important:

(A) This program requires additional information to run. For example, in the calibration folder, put a number of images from those used in training the neural network. We used 150 such images.

(B) Also, to correctly quantize the model, please modify the following line:

image = pre_process(image, (416, 416))

and insert the real resolution on which the model was trained.

(C) In quantizing_Model.py, you have a "load" function that loads all images. Inside "load" function, you have a call to "pre_process", here do all the preprocessing steps required by your real application - in this mode, the images supplied to quantization will be as similar to the ones used by your application.

In the end, if all will finish with success, you will get:

[INFO_Q]: Vitis AI Quantization...
[VAI INFO] Using func format quantizer
[VAI INFO] Quantizing without specific `target`.
[VAI INFO] Start CrossLayerEqualization...
10/10 [==============================] - 9s 993ms/step
[VAI INFO] CrossLayerEqualization Done.
[VAI INFO] Start Quantize Calibration...
4/4 [==============================] - 141s 27s/step
[VAI INFO] Quantize Calibration Done.
[VAI INFO] Start Post-Quant Model Refinement...
[VAI INFO] Start Quantize Position Ajustment...
[VAI INFO] Quantize Position Ajustment Done.
[VAI INFO] Post-Quant Model Refninement Done.
[VAI INFO] Start Model Finalization...
[VAI INFO] Model Finalization Done.
[VAI INFO] Quantization Finished.

13. In the last step, once the DNN model had been correctly quantized in int8 format, it must be compiled for our target board's DPU architecture (DPUCZDX8G-Kria KR260). To do this, we developed a new scrip program: compiling_Model.sh. The compiling_Model.sh can be downloaded from https://github.com/dmdobrea/shieldUAV/blob/main/06_YOLO_conversion_to_DPU/compiling_Model.sh. So, let's start the last step of model conversion, Model Compilation:

$ ./compiling_Model.sh KR260 yolo4_UAV_q.h5

The yolo4_UAV_q.h5 must have been placed in the ./build folder previously. Once the compilation process ended with success, you should see something like this:

[UNILOG][INFO] Compile mode: dpu
[UNILOG][INFO] Debug mode: null
[UNILOG][INFO] Target architecture: DPUCZDX8G_ISA1_B4096
[UNILOG][INFO] Graph name: model, with op num: 583
[UNILOG][INFO] Begin to compile...
[UNILOG][INFO] Total device subgraph number 5, DPU subgraph number 1
[UNILOG][INFO] Compile done.
[UNILOG][INFO] The meta json is saved to "/workspace/my_model/./build/compiled_KR260/meta.json"

[UNILOG][INFO] The compiled xmodel is saved to "/workspace/my_model/./build/compiled_KR260/KR260_yolo_q.h5.xmodel"

[UNILOG][INFO] The compiled xmodel's md5sum is 65ecbc021feb7cd00ec9e3259787bf4c, and has been saved to "/workspace/my_model/./build/compiled_KR260/md5sum.txt"

Encountered problems:

1. We were unable to get the YOLOv4tiny model. Due to the existence of "Lambda" layers unsupported by the DPU, the weights quantization tool failed. We are not saying that there is no solution, but despite all our efforts (for example applying the steps presented in the tutorial https://github.com/Xilinx/Vitis-AI-Tutorials/blob/1.4/Design_Tutorials/07-yolov4-tutorial/readme.md, but also analyzing and modifying these steps), we could not solve this problem.

2. Also, we wasted a lot of time installing all the supporting packages necessary to perform all these steps. In the end we were able to find an "innovative" solution, :-(, in which we were able to install all the support packages for subpoints 1, 2 and 3 on a Jetson AGX Xavier development board, and the rest of the support components for subpoints 4 - 13 on a I5 laptop.

Obtained results:

In the end, we obtained the following neural models capable of being deployed on the Kria KR260: YOLOv3 tiny, YOLOv3, YOLOv7 tiny, and YOLOv7.

10. UAV video tracking algorithm

Even if the YOLO object detection deep neural network has impressive detection accuracy, being a state-of-the-art object detection algorithm, its performance is not 100%. Moreover, by the quantization process of the model and/or by modifying the deep neural model so that to be compatible with the DPU, we have additional losses in object detection performances - even if they are small, 1-2-3-4%, they exist. So, there will be frames where the object will not be detected – you can see this effect in the following video in the left column of video captures when only the YOLOdetection algorithm is active. Our flying algorithm makes corrections to each acquired frame in order to intercept the aggressor drone. For this reason, if there are one, two, three, or more frames without UAV detection (in the situation where the UAV system exists), the shieldUAV will need to overcompensate when detection is finally made. In such a case, the system may achieve its proposed goal (to intercept its target) more difficultly or not at all.

During the activities carried out within this project, the object detection algorithm had 3 different implementations:

1. In the first implementation, we only used the YOLO detection algorithm. When it was integrated with the object tracking system (see the section "The object tracking software component"), the problems started appearing on the frames where the detection failed (see the following video, watching the videos from the left where only the YOLO detection algorithm is active). For this reason, it is necessary to identify a solution to improve detection accuracy. Using a more performing YOLO model was not an acceptable solution due to the low number of frames per second made by it.

2. After many attempts to solve this problem, the best approach we discovered was the use of a correlation tracking algorithm (CTA) on the frame where the YOLO model was unable to detect the object(s). The results of using this combination of methods (detection based on YOLO and a correlation tracking algorithm) at the same time are presented in the following video – the right column of videos:

In the above video, every time the detection of the YOLO neural network occurs, it is indicated by a blue rectangle with a red center. In the frames where the YOLO neural network cannot detect the object, CTA is used - detection of this algorithm is presented as a red rectangle with a blue central point.

3. Using the previously presented combination of methods, a detection cycle was around 0.175 seconds, 0.110 seconds for the YOLO neuronal model (running on the DPUunit), and around 0.060 seconds for the CTA algorithm (running on the ARM Cortex-53 processor). To further increase the execution speed of the detection algorithms, the YOLO neural model was encapsulated in a different execution thread so that the YOLO and CTA algorithms are executed in parallel - see for example the following implementation (line 254): https://github.com/dmdobrea/shieldUAV/blob/main/07_YOLOv3-tyny_UAV_plane/tools/object_center.py.

Since we will never know in which frame the YOLO-type neural network is unable to detect the tracked object, the CTA algorithm will run in parallel with YOLO. When YOLO does not detect any object, the result returned by CTA will be used. For the CTA algorithm to work correctly, it needs to detect the object from the previous frame, regardless of whether this detection is made by YOLO or by CTA. The implementation of a correlation tracking algorithm from dlib library was used. This algorithm has a significant advantage in successfully dealing with objects with large-scale variations in complex images. This situation perfectly describes our case of detecting objects placed at different distances.

By using the following links, the latest version of the UAV tracking algorithm can be downloaded:

(A) For YOLOv3-tiny: https://github.com/dmdobrea/shieldUAV/tree/main/07_YOLOv3-tyny_UAV_plane

(B) For YOLOv3, YOLOv7-tinyand YOLOv7: https://github.com/dmdobrea/shieldUAV/tree/main/08_YOLOv3-7_UAV_plane.

The existence of two different Python programs for the above versions of the YOLO neural network is due to the distinct architecture of YOLOv3-tiny, which has only two output layers, compared to the other architectures with three output layers.

In the following, a video is presented with the detection results of the YOLOv3-tiny algorithm:

We decided to use in our tracking system only YOLOv3-tiny due to its performance. YOLOv3-tiny is able to obtain 6.34 frames per second. All the other algorithms are unable to perform detections in real-time. To give you only one idea, YOLOv7 gets 0.23 frames per second.

Due to the large size of the models we trained and tested, they were posted in the following folder: https://drive.google.com/drive/folders/11QGuT2wW5nQgVnQSLV9aVMeFdSbEkN2d?usp=drive_link.

11. ROS2 (on Kria KR260 development board) – PX4 (running on HoverGames UAV) integration

11.1.Setup Micro XRCE-DDS Agent & Client

A companion computer like Kria KR60 can connect to a PX4 OS running on the FMUK66 FMU by several methods (like serial, UDP, TCP, or CAN FD). These connection methods are supported through a uXRCE-DDS middleware component, consisting of a clientapplication running on PX4 autopilot and an agentapplication running on the companion computer.

11.1.1. Start the Client (placed on the FMU – FMUK66)

The uXRCE-DDS client application is included in PX4 firmware by default.

On the HoverGames drones, we used TELEMETRY 2 (TELEM2 - SERIAL 2/TELEMETRY 2/IRDA, /dev/ttyS1) port of the FMUto make the connection between the companion computer and the FMU. MAVLink may be configured on the TELEM2 port, so we first need to disable MAVLink on this specific port and then enable and configure the uXRCE-DDSclient on TELEM2. Bu using the QGroundControl, all the settings can be done in the graphical mode. So, launch the QGroundControl.

1. First, go to Vehicle Setup, and from the left bar, select Parameters.

2. From the new column of buttons, select UXRCE-DDS Client and configure it on the Telem 2 port.

3. Next, configure the serial port's baud rate at a value of 115200.

4. In the last step, from the right-top part, use the tool button to reboot the UAV system.

11.1.2. Setup the Agent

The first step was to install the agent onto the companion computer (Kria KR260) from the source. So, in a terminal window on Kria KR260, build and install the agent:

$ git clone https://github.com/eProsima/Micro-XRCE-DDS-Agent.git
$ cd Micro-XRCE-DDS-Agent
$ mkdir build
$ cd build
$ cmake ..
$ make
$ sudo make install
$ sudo ldconfig /usr/local/lib/

The physical link between FMU (FMUK66) and the offboard computer (Kria KR260) is, in my case, a serial one. Based on this information, start the agent on my offboard computer (NavQ Plus):

$ MicroXRCEAgent serial -b 921600 -D /dev/ttyUSB0

11.2. Build ROS 2 Workspace

Now it is the time to create a workspace for the ROS2 in your home directory. As a direct result, in a newly opened terminal:

1. Create a new workspace in your working directory:

$ mkdir -p /ros2_kamikazeUAV_ws/src
$ cd ~/ros2_kamikazeUAV_ws/src

2. Clone the example repository (px4_ros_com) and px4_msgs to the /src directory:

$ git clone https://github.com/PX4/px4_msgs.git
$ git clone https://github.com/PX4/px4_ros_com.git

3. Being in the root of your workspace (~/ros2_kamikazeUAV_ws), source the ROS 2 development environment into the current terminal and compile the workspace using:

$ source /opt/ros/humble/setup.bash
$ colcon build

This builds all the packages under the /src folder. All the application we developed requires to be built in a workspace with the same uORB message definitions (placed in px4_msgs package) that PX4 used.

11.3. USB Serial Port Software setup

To avoid confusion between the two USB FTDI devices (one for the interface with FMU and the second one used to interface with our microcontroller interface unit) that have the names /dev/ttyUSB0 and /dev/ttyUSB1 alternately depending on the connection order, we create a symlink from ttyUSBx to a friendly name, related to the Vendor and Product ID of the USBdevice.

To get Vendor and Product ID of the USBdevices lsusb command was used:

$ lsusb
………………………………………
Bus 003 Device 004: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC
Bus 003 Device 002: ID 0424:2742 Microchip Technology, Inc. (formerly SMSC) USB2734
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 0424:5743 Microchip Technology, Inc. (formerly SMSC) USB5734
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 0000:0538 USB OPTICAL MOUSE
Bus 001 Device 005: ID 10c4:ea60 Silicon Labs CP210x UART Bridge
………………………………………

The USB FTDI serial interface with the FMUis:

Bus 003 Device 004: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC

and the USB FTDI serial interface with the developed interface board is:

Bus 001 Device 005: ID 10c4:ea60 Silicon Labs CP210x UART Bridge
In conclusion, we created a new UDEV rule in a file called /etc/udev/rules.d/99-kriaKR260.rules with the following content, changing the idVendorand idProduct to yours.
SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", SYMLINK+="ttyFMUK66"
SUBSYSTEM=="tty", ATTRS{idVendor}=="10c4", ATTRS{idProduct}=="ea60", SYMLINK+="ttyIOinterf"

Now, you will be sure which device (the FMU or the interface board) is connected to each individual USB port: /dev/ ttyFMUK66 and /dev/ttyIOinterf.

Before restarting the Ubuntu OS, add the current user in the tty and dialout groups using usermod. In this mode, someone will avoid executing these scripts as root.

$ usermod -a -G tty ubuntu
$ usermod -a -G dialout ubuntu

So, right now, reboot the Ubuntu OS, and from this point you can use:

$ MicroXRCEAgent serial -b 921600 -D /dev/ttyFMUK66

11.4. Testing the bridge connection

To check the bridge, see if you can receive from the drone new topics:

$ cd ~/ros2_ws
$ . install/setup.sh
$ ros2 topic list

If the bridge is working, you should see a list of topics that match the ones sent from the NXP HoverGames drone:

/fmu/in/obstacle_distance
/fmu/in/offboard_control_mode
/fmu/in/onboard_computer_status
/fmu/in/sensor_optical_flow
/fmu/in/telemetry_status
/fmu/in/trajectory_setpoint
/fmu/in/vehicle_attitude_setpoint
/fmu/in/vehicle_command
/fmu/in/vehicle_mocap_odometry
/fmu/in/vehicle_rates_setpoint
/fmu/in/vehicle_trajectory_bezier
/fmu/in/vehicle_trajectory_waypoint
/fmu/in/vehicle_visual_odometry
/fmu/out/failsafe_flags
/fmu/out/position_setpoint_triplet
/fmu/out/sensor_combined
/fmu/out/timesync_status
/fmu/out/vehicle_attitude
/fmu/out/vehicle_control_mode
/fmu/out/vehicle_gps_position
/fmu/out/vehicle_local_position
/fmu/out/vehicle_odometry
/fmu/out/vehicle_status
/parameter_events
/rosout

If the bridge is not working, you will receive the default ROS2 topics:

/parameter_events
/rosout

If there are issues, use export ROS_DOMAIN_ID=0. It is considered that only this option is hardcoded in the AP_DDS library. Also, consider disconnecting from the internet while running.

Let's read a topic:

$ ros2 topic echo /fmu/out/sensor_combined

And you will get something similar with this:

---
timestamp: 1721560868336910
gyro_rad:
- -0.01551580149680376
- -0.004595236387103796
- 0.0026286006905138493
gyro_integral_dt: 4989
accelerometer_timestamp_relative: 0
accelerometer_m_s2:
- -0.3143690228462219
- 0.021734124049544334
- -9.78078556060791
accelerometer_integral_dt: 5001
accelerometer_clipping: 0
gyro_clipping: 1
accel_calibration_count: 2
gyro_calibration_count: 1
---

Another sanity check can be done through a "listener" application from the example repository (px4_ros_com):

$ ros2 launch px4_ros_com sensor_combined_listener.launch.py

If the bridge is working correctly, you will be able to see the data being printed on the terminal/console where you launched the ROS listener:

…
[sensor_combined_listener-1] RECEIVED SENSOR COMBINED DATA
[sensor_combined_listener-1] =============================
[sensor_combined_listener-1] ts: 1721561029583935
[sensor_combined_listener-1] gyro_rad[0]: -0.00773117
[sensor_combined_listener-1] gyro_rad[1]: -0.00021979
[sensor_combined_listener-1] gyro_rad[2]: 0.000684838
[sensor_combined_listener-1] gyro_integral_dt: 5000
[sensor_combined_listener-1] accelerometer_timestamp_relative: 0
[sensor_combined_listener-1] accelerometer_m_s2[0]: -0.309921
[sensor_combined_listener-1] accelerometer_m_s2[1]: 0.0120332
[sensor_combined_listener-1] accelerometer_m_s2[2]: -9.78081
[sensor_combined_listener-1] accelerometer_integral_dt: 5003
…

Encountered problem:

The most "painful" problem (from the financial point of view but also the completion of the project) from the entire project was the destruction of two FMUK66 type FMUs (approximately 220 USD/per piece). A faulty USB FTD module destroyed the first FMU on the drone it was on, and when this FMU was replaced, the second one was also destroyed. By the time I realized that the interface module was heating up, it was already far too late:

Now, a video from the double funeral:

12. Offboard control

Having now a functional bridge between the FMU of the shieldUAV drone (running PX4) and ROS 2 (supported by Kria KR260) was the first step in offboard control.

Real-time control of a drone's position can be achieved in several different ways – by controlling: the position, speed, and acceleration or by providing GPScoordinates. However, considering the proposed objective, we only analyzed the first two ways of controlling the shieldUAV drone: by position and speed.

The codes for these two control modes can be downloaded from the following links:

(a) Control through velocity: https://github.com/dmdobrea/shieldUAV/tree/main/09_offboard_velocity,

(b) Control through position: https://github.com/dmdobrea/shieldUAV/tree/main/10_offboard_position.

For the PX4 autopilot to be able to obey a specific set point (velocity, position, or attitude), several requirements must be met:

• The PX4 must be in Offboard mode – to enter this mode, the drone must be armed (… and to be armed other sets if conditions must be satisfied);
• The setpoints must be streamed at a rate greater than 2Hz before entering the Offboard mode, and while this mode is active. From this requirement, someone can understand better the necessity of using the call of self.newTrajectorySet(0.0, 0.0, 0.0, 0.0, False) when the drone is not in the Offboard mode.
• The changing command between position and velocity is mutually exclusive. If you choose to specify a velocity setpoint, all uncontrolled setpoints (e.q., position) must be set to NaN - but all of these require a float value, so use float("nan").

To view the way of controlling the shieldUAV drone according to the velocity mode (movement on a circular trajectory), please watch the following video: