AI powered Surveillance Robot

👮 Surveillance and patrolling robots are increasingly being used in various sectors for security and monitoring purposes. These robots can operate autonomously or semi-autonomously and are equipped with various sensors and technologies to perform their tasks effectively.

Lets take detailed overview at their 1. features, 2. applications, and 3. benefits:

1. Features ⭐ - Existing Autonomous surveillance Robot systems have

🚙 Mobility:

• Wheeled Robots: Suitable for smooth terrains like indoor environments.
• Tracked Robots: Better for rough or uneven terrains.
• Legged Robots: Provide high mobility over obstacles and stairs.

📷 Sensors

• Cameras: For visual monitoring and recording.
• Infrared Sensors: For night vision and thermal imaging.
• Ultrasonic Sensors: For obstacle detection and avoidance.
• LIDAR: For precise mapping and navigation.
• Microphones: For sound detection and recording.

📞 Communication:

• Wi-Fi: Common for indoor environments.
• Cellular (4G/5G): For wide-area coverage.
• Bluetooth: For short-range communication.
• Mesh Networks: For robust and reliable communication between multiple robots.

🤖 Autonomy:

• Autonomous Navigation: Using SLAM (Simultaneous Localization and Mapping) to navigate and map environments.
• Manual Control: Remote control options for human operators.
• AI Integration: For advanced decision-making and anomaly detection.

🔋 Power Supply:

• Batteries: Rechargeable batteries with varying capacities.
• Solar Panels: For outdoor robots to extend operational time.

2. Applications - Potential applications of such systems are ubiquitous

🚨 Security and Surveillance:

• Intrusion Detection: Identifying unauthorized access and raising alarms.
• Patrolling: Regular patrolling of predefined routes to deter intruders.
• Crowd Monitoring: Monitoring large gatherings for safety and security purposes.

🕵️ Inspection:

• Infrastructure Inspection: Monitoring structural health of buildings, bridges, and pipelines.
• Environmental Monitoring: Checking for hazardous conditions in industrial settings.

⚔️ Military and Defense:

• Border Patrol: Monitoring and patrolling border areas.
• Reconnaissance: Gathering intelligence in hostile or dangerous areas.

🩺 Healthcare:

• Hospital Security: Monitoring patient areas and ensuring safety.
• Sanitization: Robots equipped with UV lights or disinfectant sprayers for sanitizing spaces.

🛒 Retail and Hospitality:

• Customer Assistance: Providing information and assistance to customers.
• Stock Monitoring: Keeping track of inventory levels and product placement.

3. Benefits - By taking this Robotic assistance would definitely beneficial

• Cost-Effective: Reduces the need for human patrols, lowering labor costs.
• Efficiency: Robots can operate 24/7 without fatigue.
• Safety: Reduces risk to human security personnel, especially in dangerous environments.
• Data Collection: Continuous data collection for analysis and improvement of security protocols.
• Adaptability: Robots can be reprogrammed for different tasks and environments.

Now lets see, in case of security why we really need assistance of AI based Surveillance Robots in our daily life

🚩 Problem Statement

Terrorism is a global problem that affects people of all ages, races, and religions. It is important to remember the victims of terrorism and to work to prevent future attacks. It is estimated that over 57,000 people have been killed in terrorist attacks since 2001. This number is likely higher, as many attacks go unreported or are not properly documented.

In India, 2008 Mumbai attacks, 10 Armed militant entered into Hotel, Railway Station, Hospital and other crowdy places and killed 175 peoples, and injured more than 300.

What if we could quickly detect suspicious peoples carrying guns when they landed on beach? We would have definitely saved a lot of lives.

🙋 Proposed Solution

I want to build AI and Deep learning based Surveillance/patrolling Robot with 360 degree PTZ and HD cameras mounted on it, which will move around in these places (like Bus, Railway, Tram, Subway Stations, Airports and other public places) autonomously and (with use of Deep learning and AI video analytics Techniques) to keep looking for “Suspicious activities like People carrying weapons, Guns etc” and warn the security personals for any such sightings.

I want to train and make this robot Smart enough, so that it can identify Suspicious activities like People carrying weapons, identifying violent individuals. It should also be able to identify and differentiate between security personals and possible threat. This robot will also provide information and Warning signal the authorities about any sightings of Wanted Criminals who’s blacklisted by Police, (criminal database training should be done before).

🤖 Existing Solutions / Commercial Products

There are several existing AI Surveillance Robot applications around world. Employed by NYPD, UK Police, Singapore Police, Japan and South Korea Police etc. In India there are very few examples of full featured Police Robots deployed. One notable example is the deployment of a humanoid robot named KP-Bot by the Kerala Police in 2019. This robot is primarily tasked with handling non-emergency tasks at police headquarters, such as providing information to visitors, directing them to relevant departments, and collecting feedback.

Popular Examples

Knightscope K5: A fully autonomous security robot used for outdoor patrolling.

Boston Dynamics Spot: Very popular, A versatile quadruped robot used in various applications including security.

Robo Guard by Singapore Police: A mobile surveillance robot equipped with cameras and sensors for security patrols.

My solution will somewhat different, this Robot will be equipped with cameras and sensors can patrol public areas, monitor crowds, and detect suspicious activities, providing real-time information to human officers. Moreover Operation officer can manually move the robot overriding the Automation when necessary.

💕 Thank You Hackster for organizing and hosting this amazing "Getting Edgy with Machine Learning Contest" and Big Thanks to Infineon for supporting my project idea by sponsoring/providing their amazing Hardware AMD KRIA™ KR260 Robotics Starter Kit

How does our solution will work?

I want to build 4 wheel Bot, which will consist of One PTZ camera and 2-4 HD IP cameras mounted continuously provide video stream (RTSP). I’ll try RTSP Feed Camera Source for Edge Impulse on Kria KR260.

I have seen nice tutorials/Articles by *Whitney Knitter* on *Hackster*. I’ll try to use her ‘Lego Land Rover Defender Robot’ as a reference for my design. But my Edge Impulse model will be trained and optimized for detection of suspicious/Violent activities and Weapons detection. I have made one similar kind of autonomous Robot using Jetson Nano and JetBot Kit, but its camera was at steady position. Here in this design I will try to use PTZ and IP camera and (RTSP feed).

I will use assembly of that Robot like motors and Chassis etc. In terms of raw performance as per specs, It looks the NXP Kria KR260 is more powerful than the NVIDIA Jetson Nano, offering up to 2.6x times the performance (Up to 4 TOPS).As for Kria KR260 we can use native ROS 2 and Ubuntu support with Kria Robotics Stack (KRS).

For ML I will mostly going to prefer Edge Impulse. Once RTSP Camera Integration in Edge Impulse done we can then collect frames. Security camera can be accessed via the RTSP link, and it works with OpenCV in Python, the final step is to point the Edge Impulse image classification code to use it as the image source. Then, There will be different ML models to be trained, like free roam, Obstacle avoidance and then Suspicious activity detection.

Please note that, I want to keep speed of Robot bit slow, so it will be more accurate in obstacle avoidance and detection of peoples with Suspicious activities. This robot will inform Authorities according level of threat detected. Like suspicion, Low, moderate, high etc. It will be programmed to record the activity with video/ photos/frames/Images of (for example any person carrying gun). It’ll be two level process, The operator of this Robot will get real-time notification about any Suspicious person sightings, then he/she will diagnose further using recorded frames and can manually detect any threats.

Current Progress -

Unboxing -

AMD KRIA™ KR260 Robotics Starter Kit came with following components

• Kria KR260 Robotics Starter Kit (Kria SOM + Carrier Card + Thermal Solution)
• KR260 Power supply and Adapter (12V, 3A) with Universal converters
• MicroSD Card 64 GB
• USB-A to micro-B Cable
• CAT5 Ethernet Cable

I started with Getting Started Guide given by AMD. I just need following component to get started

• USB Keyboard
• USB Mouse
• DisplayPort™ Cable (for connecting to a monitor)
• DisplayPort Display

I was unaware about 3rd and 4th requirements. I have Acer HDMI based Monitor. Hence to connect KRIA board and get started I either have to] purchase new DisplayPort Monitor or purchase DisplayPort to HDMI Adapter. So I decided to purchase DisplayPort to HDMI Adapter.

I tried 4 different Active DisplayPort to HDMI Adapters but none of them seemed to work

Then I decide to go with Active DP to VGA adapter. I bought VGA Display cable and purchased following adapter. This time I got success in booting Ubuntu 22.04 which is recommended on Getting started page

But before booting Ubuntu 22.04, I followed following guide to update the system firmware

To enable complete board functionality, compatibility with latest Operating Systems, and best performance, be sure to install the latest AMD provided boot firmware following the firmware update instructions available on the K26 Wiki.

And following binary file

Basically to do all above I followed this guide. But used above binary update file.

=================================================================

Projects Whole Assembly (proposed)

Assembly (current Status)

Robot Chassis

I have JetBot chassis, I am planning to modify it to get it work with KRIA KR260 and ESP32. As ESP32 support MicroROS. So we can control motors via Motor driver and ESP32 MicroROS. Then we will connect KR260 to ESP32 serially. KR260 will be having ROS2

***I unable to get work with I2C based driver board on JetBot Chassis. So I decided to go with typical set-up of L293D Dual H Bridge Motor driver.

Connected ESP32 via UART with KRIA KR260. Currently I am powering driver from ESP32 but there will be dedicated 5V external power supply (LIPO battery pack) for motors.

Camera

I do not have High speed 4K camera to interface with KRIA board. I decided to use my C270HD webcam for starters. There will be one PTZ camera mounted on Robot . Tapo C500

PNYQ Installation:

Refer to the official PYNQ repository (amd/Kria-RoboticsAI) for detailed instructions on installing PYNQ-DPU on the Kria KR 260 Robotics Starter Kit.

This guide covers the installation process and also includes information on setting up ROS2.

The Virtual environment which we are going to set during PYNQ and ROS 2 installation will also be used for MicroROS.

MicroRos Installation

After installing PYNQ, we'll need to set up MicroROS on our ESP32 to establish serial communication between the Kria KR260 and the ESP32. MicroROS is a lightweight implementation of ROS 2 designed for resource-constrained devices like ESP32.

PS2 Controller for Robot

I want to interface following PS2 controller for the Robot. I have interfaced it successfully with my JetBot setup. I hope it will work with Kria KR260 also

WiFi Adapter -

I have purchased EdiMax Usb Wifi Adapter for connecting KRIA KR260. I need to test it its working. I hope it will work. This will make KRIA kit headless.

Last Update : 20/07/24

- Project is currently in development phase.

- Basic Object Detection is completed successfully.

- Working on Open source Weapon Detection database.

- HostPC (Laptop) running dual boot Ubuntu+W10 got corrupted..😖😔

- Restored PC, RSTP Stream with PTZ camera TapoC500 working. Need to post the video.

Future:

This project is a rigorous evaluation of my capabilities, persistence, and expertise, forcing me to grow and adapt. This contest is proving to be a great learning experience. Developing this project great experience so far. I will definitely continue to work on this even though contest is getting over.