Drone Patrol for Social Distancing

The Goals:

Over the past year, the world has been at a standstill due to the Covid-19 pandemic. Most of the concern about people has been about not social distancing or doing their part to stay home. Although we can’t enforce stay at home rules with the help of UAV’s, we can certainly try to monitor social distancing in places of high public traffic, such as beaches. Beaches are prone to visitors no matter the time of year or the pandemic conditions. People continue to travel to beaches, with many news outlets expressing concerns of thousands of people flocking to coastlines despite the growing number of cases of coronavirus in California. Our project aims to use UAVs and integrated algorithms to tackle the monitoring of social distancing along these beaches.

AuroraNautics, Texas State University, and Airogistic have combined teams to submit a final report on the NXP Hovergames 2 sponsored project. The objective of our team was to combine team resources and expertise to accomplish three primary technical objectives:

Utilization of the NXP PX4 Hovergames Drone and the NAVQ companion electronics to advance our research and development of a complete UAV drone operations solution. We envision a fully autonomous system solution for drone flight operations including drone pads equipped with predictive diagnostics, an operation center for AI weather analysis, and drones packaged to operate in a wide variety of environmental conditions flying missions 24/7. (Texas State U EE/ME and Airogistic)

• Utilization of the NXP PX4 Hovergames Drone and the NAVQ companion electronics to advance our research and development of a complete UAV drone operations solution. We envision a fully autonomous system solution for drone flight operations including drone pads equipped with predictive diagnostics, an operation center for AI weather analysis, and drones packaged to operate in a wide variety of environmental conditions flying missions 24/7. (Texas State U EE/ME and Airogistic)

Drone assemblies that include AI visualization algorithms based on the NXP FMUK66 PX4 flight control and integrate the Hovergames 2 of the NXP NAVQ companion processor boards base on the i.MX8 mini embedded machine learning and artificial intelligence processor board that was shipped with the Google Coral AI camera. Our team has developed software for the purposes of visual monitoring public beaches (or public parks) to determine the beach/park conditions relating to COVID-19 recommendations for public distancing in order to prevent overcrowding and maintain safety in spacing. (AuroraNautics)

1.Drone assemblies that include AI visualization algorithms based on the NXP FMUK66 PX4 flight control and integrate the Hovergames 2 of the NXP NAVQ companion processor boards base on the i.MX8 mini embedded machine learning and artificial intelligence processor board that was shipped with the Google Coral AI camera. Our team has developed software for the purposes of visual monitoring public beaches (or public parks) to determine the beach/park conditions relating to COVID-19 recommendations for public distancing in order to prevent overcrowding and maintain safety in spacing. (AuroraNautics)

2.Creation of a combined drone operations decision, reliability, and information portal. The portal includes public web pages that manage flight operations and weather as well as offer information to help alert people and authorities about the beach/park conditions in the future. The concept is to aid public or park goers in decisions as to which beach or park is safe. The user interface design is architected to be simple with color-coded ratings as to whether a day trip is appropriate for flight and also allows set expectations before people leave for a visit so that public social distancing is maintained during COVID-19 restrictive challenges. ( Texas State IE and AuroraNautics)

3.Creation of a combined drone operations decision, reliability, and information portal. The portal includes public web pages that manage flight operations and weather as well as offer information to help alert people and authorities about the beach/park conditions in the future. The concept is to aid public or park goers in decisions as to which beach or park is safe. The user interface design is architected to be simple with color-coded ratings as to whether a day trip is appropriate for flight and also allows set expectations before people leave for a visit so that public social distancing is maintained during COVID-19 restrictive challenges. ( Texas State IE and AuroraNautics)

The work on this HoverGames project was performed by undergraduate students, drone industry experts, and drone entrepreneurs. Our team combines electrical, manufacturing, industrial, and aerospace engineers located in San Marco & Austin, Texas, and San Jose & Sacramento, California. The work effort encompassed a wide range of engineering tasking and objectives. Overall our mission and efforts are focused on creating a commercially viable end or end solution that can be deployed and operated in beach areas and communities worldwide.

While some of the engineering tasks had little to do with software or AI code on the NavQ, all tasks contributed to our project. All of the tasks were deemed necessary to create a world-class solution that is capable of operating under real-world environmental conditions while maximizing the safety of the public and validating the drone operations using automated processes in order to ensure the uptime of the service and allowing for a continuous flight of drones for this important application.

Hovergames Drone Diagnostics & Launch/Land Pad:

The creation of a solution for social monitoring on beaches or urban public areas such as parks will require more than drones and pilots. For that reason, we engaged this effort with two teams.

As part of our effort to design an effective launch/land platform to be used in picking up and delivering COVID test samples, we quickly realized that ensuring safe autonomous flight was the biggest priority. To tackle this, we decided to install a variety of sensors on the launch/land platform that could be employed to check the overall health of critical drone components before flight. A complex analysis of this sensor data as well as flight logs gave us the ability to effectively determine if a drone is flightworthy.

Sensors include RPM sensors, temperature sensors, and thrust sensors. We used the NavQ as the processor board that all of the sensors connected to. Connections needed include 5x USB, 4x GPIO, I2C, and ethernet.

To perform the preflight test, the drone would land on the launch/land platform and a mechanism would move the drone to the center of the platform. The drone would then couple with the thrust sensors so that it can take samples and the remaining contactless sensors would move into position to begin taking samples as well. The preflight test would start and commands would be sent over the telemetry radio using the mavlink_shell.py python program. The NavQ sensor module would then begin taking samples from the 3 sensor groups. At the same time, the flight logs would be sent over the telemetry radio using the mavlink_ulog_streaming.py python program and then are analyzed to check for battery health and inertial sensor functionality. These python programs are both part of the PX4 Tools made available by NXP. The vibration is also measured by the accelerometer and analyzed to check for excessive shaking, which could indicate motor, propeller, or structural failure. Preflight test results are then forwarded to the user via email.

Implementation of the Navq:

Our team’s mission focuses on social distancing detection integrated into our UAV. To achieve that goal, we need a very powerful machine onboard our drone to handle intensive image/video processing. The NAVQ board offers essential elements that our team is looking for. The powerful onboard processor and fast DDR4 memory enable us to utilize demanding hardware applications. We are not aiming to utilize the NAVQ to fly our drone but to use it for advanced image processing. We also utilized other attached hardware that comes with the NAVQ module. The high-quality camera on the NAVQ could be used for image processing enabling us to capture high-resolution images even high up in the sky. Also, the included WIFI and Bluetooth connection is something that we are utilizing for this project. The initial vision for this project is to stream live video so that the social distancing violations can be overseen by appropriate individuals such as lifeguards or law enforcement. The WIFI and Bluetooth capability on the NAVQ can stream video at high speed over long distances. The onboard quad-core arm CPU, full HD google coral camera, and high-speed wireless connections enable our team to produce real-time image processing.

Design Changes and Considerations:

Texas State/AuroraNautics HoverGames Drone Cover Design Iterations:

Climate conditions, similar to rain, fog, snowfall, or high humidity, are common. If the goal is to support continuous and automated flight operations for drones, one needs to fly with covers that protect the electronics and batteries from moisture and condensation which can cause short circuits or electrical failure of key components. For this reason, the Hovergames drone cover design was initiated to prevent water ingress during flight operations. Several design considerations were weighed. Whether the cover should include air vents, the packaging of the cover for the placement of the NavQ, and the camera were things that were considered. The effect of the cover mass on flight control and drone propulsions turned out to be critical. Most importantly the cover was designed to prevent water from directly infiltrating and damaging the core electronics: PX4 Robotic Drone Flight Management Unit (FMU) - RDDRONE-FMUK66 and RDDRONE-8MMNavQ "NavQ" Linux companion computer platform with Vision.

The cover does protect the drone electronics from rain and high wind weather during the flight operation. It was developed in two stages and worked on by two separate teams. The drone cover’s first prototype that was designed by the Texas State Manufacturing Engineering team is shown below.

The first prototypes to support the drone operations were manufactured with 3D print manufacturing using PLA, ABS, and PETG. Early prototypes of the cover were heavier than expected and the added weight exceeded 300 grams. In addition to the design being too heavy the aerospace engineers and AuroraNautics identified non-aerodynamic features that were likely to cause drag during flight. The drone with the first prototype was unable to take-off at 100 percent Hovergames throttle/power. This led the combined team to make another pass at the cover design. A new and improved prototype that was lightweight (carbon fiber) and able to handle drag forces during flight operations were designed in a 2nd design iteration by the Aerospace engineers at AuroraNautics.

It is essential to highlight the center of gravity when designing any drone. The second prototype design, taken on by the AuroraNautics San Jose team, was given top priority in integrating the drone cover to keep the overall added weight below the targeted 100 grams. The drone included additional features of the NavQ camera mounting on the front of the drone for image processing during the patrolling mission on beaches to detect people’s color signature and distance during the Covid19. The new drone cover design features the front nose, where the NavQ is mounted for accurate imagery of environmental conditionals at the beaches. The second prototype depicted. encompasses an aerodynamic friendly Hovergames drone cover design that can protect the drone during flight operations and can fly with material less than the targeted 100 grams providing superior aerodynamic performance.

Test Flights:

Multiple test flights were completed throughout the course of the competition to ensure that the drone was fully operational as changes were made. Initial flight testing was performed inside of a specially designed enclosed structure that helped to reduce fatal crashes and facilitate rapid development. Before beginning any test flight, all inertial sensors were calibrated through QGroundControl.

The first test flights consisted of hover and pitch/yaw/roll testing on all axis while in manual mode. Next, position and altitude flight modes were tested, which require more inertial sensors than manual mode does. At this point in the testing, the drone was very unstable and jerky in flight. To fix this issue, we downgraded the FMU firmware and copied the parameters list from another working HoverGames drone.

Once the drone was stable and we were confident it was operating correctly, we installed the initial prototype of the weather cover. It was very quickly obvious that the propulsion system of the drone would need to be upgraded in order to be able to fly with the added weight of the NavQ and the weather cover. We are very close to performing a test flight with our next iteration of the weather cover.

Launch Land Pad Iterations:

The launch land pad iterations include using the temperature sensor to detect when a motor overheats and uses a force sensor to measure motor lift force. It also ensures that the appropriate lift is generated. This is crucial because it allows the individual controlling the drone to be aware of the health of critical drone components. The temperature sensor and force sensor are attached to the launch land pad. Therefore, when a drone prepares its flight mission, a preflight test is performed prior to flight to ensure that these components are performing.

Weather Stations & Flight Decision Portal:

The team developed a dashboard to serve as a portal to expose weather critical operating parameters and key performance indicators to provide a fly/no-fly decision at the beginning of a flight. This dashboard parses weather data served from localized micro weather stations hosted by ranchers, homeowners, and businesses. The Texas State Skywalker IE team derived a distance algorithm using the GPS mapping coordinates of the flight path entered in a QGroundcontrol mission.

Note: Red indicates no-fly conditions

The next step for the portal and dashboard is to begin implementing the mechanical pre-flight test analytics into the fly/no-fly decision matrix along with visualizing them on the platform. With the advancement in UAVs, Urban Aerial Mobility (UAM), and new operational models in FAA regulated airspace, smarter airports working for aircraft operators are needed to standardize the availability of localized weather and flight hazard information. In order to maximize flight operations uptime, and enhance safety, the creation of a new type of aircraft and weather dashboard or portal to provide fly/no-fly recommendations at the beginning and during flights for aircraft operators is pivotal. The goal is to ensure each individual flight’s success. Considering the accessibility to the location of aircraft departures both at airports and at operators’ locations, the readings from micro-weather stations along the flight path provide conditions necessary to determine flight suitability. The entirety of this information and resulting verdicts needs to be accessible in an app such as FAA’s B4UFly. Or a new application for aerial applications similar to WAZE™.

Navq mounting:

A combined team effort was made to determine the optimal mounting of the NAVQ and the Google Coral Ai camera on the Texas State and AeroNautics Hovergames drones. Both teams elected to mount the NAVQ and the camera in the front nose cone area of our Hovergames drone design. Because the Texas State team was focused on launch and landing support for continuous drone autonomous operations, it made the most sense for their team to mount the camera face down. Contrasting for the beach application a forward or mostly horizontal mounting of the camera orientation was desired by the AuroraNautics team.

The two different camera orientations are depicted.

For the future, AuroraNautics has identified the HoverGames drone has a requirement to utilize both NavQ camera orientations for the separate functions of application AI (front) and launch/land alignment with the down-facing vision. This could be accomplished for the disparate smart vision requirements by mounting the camera on a gimbal that rotates the Google Coral AI camera abound the NavQ board 90 degrees. Elegantly, a design that separates these functions by supporting both a front-facing and down-facing camera interface to the NAVQ for future NXP based Hovergames drone application designs would be ideal.

Iteration 1; Navq Social Distancing Computer Vision :

The initial idea was to utilize color imaging for person detection and use various colors to indicate ranges. The idea is that each person detected would be colored differently based on distances. For instance, people in close proximity would be colored red, while people being far would be blue. To accomplish this, we started using OpenCV and python. OpenCV offers a lot of tools that we can utilize for coloring. We first set up VideoCapture from the OpenCV library to receive video through the coral camera in our code. With the correct pipeline for the VideoCapture, we can pass each frame of an image taken by the camera through our color filtering functions within python. We wrote the python script that utilized the OpenCV built-in function cvtColor to reassign the color image taken to a grayscale image. Grayscale images are much easier to process than RGB images, simply because grayscale images only deal with integers instead of arrays of values. With the image in grayScale, we can run through another function called bitwise_and to include the captured person in a separate frame. Based on our color definition, each person detection would be assigned a specific color. However, the downside of this method is how graphically intensive the workload is. The NAVQ has to process images for detection while giving color accordingly. Using this method would result in slow system response, thus not good enough for our project. Another issue with coloring detection is background color. Since our drone aims for beach applications, the sand and ocean's reflection can cause a significant problem.

Aerial Color/Thermal Beach Image:

Iteration 2; Region Based Convolutional Neural Network (R-CNN):

Another iteration of our object/human detection software is based on a pre-trained neural network model from MobileNet. The object detection uses a fast-R-CNN algorithm to analyze each image being processed. The algorithm loops through the images multiple times and acquires individual pixel values. With the information of individual pixel values, the algorithm then assigns vectors in groups of pixels, creating a vector flow field in the process. Finally, with the vector field information, the algorithm passes it through multiple layers of neural networks with about 256 neurons per layer. The pre-trained model with weights and biases determines objects within each image and feeds it back to the main script with arrays of information. The algorithm outputs a couple of distinctive data types, containing information about each detection's level of confidence, location of detection, and what it thinks the object is. All this information gets feedbacked to the main loop and is assigned as arrays. In the main script, we used a built-in neural network function in the OpenCV library, dnn_DetectionModel. This function creates and builds a model based on the weights and configuration files we input. This model initially has a lot of promising results and is able to support real-time frame rates. However, the MobileNet R-CNN model is not supported on Ubuntu's version of OpenCV.Iteration using R-CNN

Iteration 3; You Only Look Once (YOLO):

Due to the R-CNN model's compatibility issue with OpenCV when using it on Ubuntu 20.04, our team had to find another way for the object detection model. Luckily the YOLO algorithm from Darknet offers a much more well-developed object detection model. Using the Coco data model, when using YOLO, we can reach a higher frame rate on the NAVQ. YOLO stands for you only look once. The algorithm itself will only scan the entire image once. Other detection models like R-CNN loops through the whole image multiple times, where YOLO only loops through the image once. Since YOLO only scans through the image once, it has much higher efficiency and even works on tiny computers. The YOLO object detection model works differently than the R-CNN method. It was the first of its kind when it first appeared. Instead of looping through the same images repeatedly to assign a vector field, it divides the images into sub-regions. For example, when a 1920x1080 image passes through a YOLO detection model, it gets divided into subregions, and these subregions get divided based on the grouping of pixels. If the image's pixel color contains patterns that might look like an object, it will get grouped and processed together. This way instead of looking at single pixels, the algorithm processes groups of pixels together to determine detection. Not only is this method faster, but it is much more accurate. The only issue with this model is the computing power it requires since it heavily utilizes neural networks for detection, requiring a high level of graphical computing power. In the case of our Hovergames project, we will use the TINY version of the YOLO model. Since the NAVQ board doesn't have a dedicated or integrated graphics processing unit, the TINY version can allow the software we write to run a little faster.

Iteration 4; Haar Cascade Classifiers:

Other than the object detection models we found on the internet, we also tried to utilize haar cascades for object detection. Haar Cascade classifiers is a deep learning model for effective object detection. It trains its model through a series of positive and negative detection images. The classifier uses these positive and negative images to perfect their model to detect other photos. We were hoping this model can provide a more acceptable detection over long distances. Since the model contains a lot of variation, we tested out these variations to find the best one for our project. Besides, the haar cascade model only focuses on detecting one thing or few things in each trained model, unlike the YOLO algorithm trained on the coco model containing over 100 classes. The use of haar cascade was super simple and easy. From open-source Github, we obtained a pre-trained model in XML files. Since OpenCV integrates with Haar Cascade, we need to recreate the trained model in OpenCV and ensure we include all the weights and biases information. Using the cv2.CascadeClassifier function, we included the XML file to set up our model. The second step was to convert our images from RGB to Grayscale since the omission of this step would dramatically increase the processing time. The next thing to do is obtain images from the google coral cameras to pass through to the haar cascade model. The model will spit out crucial information-containing regions of detection, and with this information, we can use cv2.Rectangle to demonstrate detections over our images. This model's advantages over YOLO are its efficiency in detecting full bodies even at some range, with YOLO being good at detecting proximity images.IterationHaar Cascade

Social Distancing Function Definitions:

With object detection setup using the various algorithms, we can create multiple functions that we can use to illustrate object detection and social distancing. There are three functions in total that we utilized in our main script, distanceCalculator, angleCalculator, horDistanceCalculator. These three functions used together in the correct order can effectively calculate each detections' position, in this case, people.

The distanceCalculator utilizes the information outputted by the detection model and the specific camera focal length to approximate a person's distance from the lens of the google coral camera. This function was first tested on and fine-tuned using real-world images to calculate some of the functions' baselines. We first took photos of 11*8 inches paper at a distance of 3 feet from the camera. Then used findContours in OpenCv to get the exact measurements of the paper in terms of pixels. With the information of pixel width of the paper and the paper's real-world size, we can then calculate the google coral camera's focal length to be about 584.47 using equation 1. Looking at equation 1, We know that the distance of an object from the camera lens can be calculated using the object's pixel width, the object's width in real life, and the camera's focal length. If we assume that an average adult is about 38 inches, we can then utilize all the variables to find the person's distance with respect to the camera. The distanceCalculator function itself requires a boxWidth input. This input is the width of the person detected by computer vision. The function returns distance as its output.

FocalLength = (Pixel*Distance)/Width Equation (1)

Another essential function is the angleCalculator function that is heavily utilized in our social distancing program. This function requires four separate inputs, location of the person in the frame, the width of the person in pixels, resolution of the picture, and field of view of the camera horizontally in degrees. With these four inputs, the function effectively calculates the centroid of the person horizontally, then compares the centroid's location to the centerline of the frame. Since we know the total pixel width of the entire frame and the camera's full field of view, we can then calculate how many degrees of variation per pixel. Finally, the difference between the centroid of the person multiplied with degrees per pixel will allow us to find the person's angular deviation from the center of the frame.

The last function we wrote was to fit the final piece of distance calculation. Although we obtained information about the distance of a person and angle, we were still unsure about the horizontal distance between the person's centroid and the image's centerline. The function horDistanceCalculator was designed to tackle this task. Since we have the information of the vertical distance and the angular deviation, we can use simple geometry to calculate the person's horizontal distances. The horDistanceCalculator takes in two arguments, vertical distance and the angle of interest, then returns horizontal distances.

These three functions mentioned above are heavily utilized in the while loop where the actual social distancing calculation occurs. In the while loop, a single frame is read and saved to a variable. Then we run this image through our R-CNN object detection model using the Detect function for the dnn_detectionModel. The Detect function returns three 3D arrays containing the information discussed above. Since the returned data is in a 3D array, we needed to run a nested for loop to decouple the matrix and access each element individually within the while loop. Each accessed element is then processed through the three separate functions mentioned above. Lastly, within the for-loop, all the calculated data will be stored in multiple different local arrays. There are two for-loops within the main while that operates the majority of the program. After the first for-loop retrieved data about detection information, this information is then processed again in the second for-loop.

The main job of the second for-loop is drawing rectangles around each person within a single frame. But to implement social distancing rules, we need to utilize the distances we calculated before. The three separate arrays, vDA, hDA, and angleHA were created to fulfill this task. These three array variables are looped through using a simple for-loop to extract information. Within the for-loop, we will calculate the distance between two detections (person) using the simple Pythagorean theorem. Then the distance will be compared with all the other detections within the array. If at any point the calculated distance compared with other detections is less than 72 inches(6 feet), the program will convert the global variable boxColor to yellow. The global variable boxColor contains color information about the rectangular boxes that will be drawn on the person. The default color for boxColor is green, but when two objects are within 72 inches of each other, the boxColor will be yellow.

The final iteration of the prototype:

Through many iterations, we tried there is not a single one that can perform the job correctly on its own. While the haar cascade full-body detection works well over long-range distances, it suffers at close range. Also, accurate distance tracking is challenging if the drone is high up. The lighting and reflection would cause significant interferences when distance tracking is operated at a high altitude. On the other hand, the YOLO model is very efficient in tracking close range(it doesn't require the person to show full body). And the tiny YOLO model provides fast processing time and improved frame rate compared to other object detection models. To have the best of both worlds, our team believes that utilizing both detection models would be beneficial and improve our patrolling drone's overall efficiency. While patrolling over the beach, the system would mostly run on the haar cascade model to effectively detect people and identify large groups. Within a close range, the drone can switch gears to the YOLO model and calculate distances between people. Our final iteration of the social distance drone will utilize both these two models discussed above to maximize our efficiency and accuracy.

Issues with the code implementation and how to overcome them:

The biggest issue with our python script was the distance calculation. Since we only used one camera to perceive the real world, the amount of data we can gather is minimal. From a single 2D picture, we can't detect accurate distances in a 3D environment. One way to tackle this issue is to find patterns inside the chaos. Knowing that an object would appear bigger when closer to the camera, we can use the google coral camera to collect images of a person at various distances away from the camera. From the data sample, we can generalize a relationship between a person's size within a picture and its respective distance. From that, we can have a very good approximation of a person's distance from the camera.

Another challenge we faced when perfecting our script was an issue with object detection. It is hard to find an open-source neural network that has high accuracy and efficiency. Most of the object detection models we tried were only good at detecting people under certain conditions. With how quickly the environment changes while a drone is flying, most of the models we found failed to detect people. The few usable ones we found were accurate but lost range or lost efficiency at close range. To combat this issue, we had to utilize multiple object detection models to increase the overall efficiency and accuracy, ensuring that our drone can work in various scenarios.

Future Work:

• SBUS communications on NavQ.
• Drone Auto Calibration as part of pre-flight.
• Resolve NXP PX4 firmware 1.11 unstable flight control
• Add Gimbal stabilization and rotational control for Google coral
• NavQ dual camera interface for down-facing and forward-facing cameras.
• Auto-actuation of the sensor heads
• Replace RPM magnetic sensor with optical RPM non-contact sensor
• Train custom neural network model for directly overhead people recognition
• Integration of maintenance diagnostics into the preflight decision portal
• Merge Video analysis into preflight decision portal
• Additional public webpages for beach patrol information