AI Juggling robot

This project has been created in attempt to create a robot capable of juggling various items without human intervention using artificial intelligence. In an ideal scenario, the AI system will make the decision of the type of object juggled from the trained data and adjust the corresponding actuative force to keep the object juggling.

Hardware setup

The project is started with a simple prototype consists of two ball launcher controlled with a servo motor respectively. The heart of this system is the Kria KR260 robotics starter kit which is a powerful platform that can manage real-time ball tracking using a webcam and actuation through servo controller running I2C.

Software

For object detection, I utilized YOLOv4 model, as it seems to have high speed and accuracy in detecting objects from the real-time video feeds from the webcam. For predicting the future positions of the balls, a Long Short-Term Memory (LSTM) network is implemented as it is suited for sequence prediction tasks.

Outcome

Unfortunately, there are many challenges that prevented the success delivery of the project. Firstly, the model trained for ball tracking is inaccurate as the model struggled to generalize across different scenarios, often predicting paths that deviated from reality. Besides that, the complex control of balls through air encountering various forces like air resistance and spin, created more uncertainty to the system.

Future Work

Despite failing to achieve a fully functioning juggling robot, I have gained a lot of experience and insights into the complexities of integrating AI with robotics and using custom FPGA applications. Even though the competition has ended, I am committed to continuing the development of the juggling robot. The lessons learned from this experience will be used to guide my future efforts. Here are the key areas I plan to focus on:

1. Improving AI Model Robustness: I will work on enhancing the robustness of our AI models by training them on more diverse datasets. This will help the models generalize better to different juggling conditions and ball characteristics.

2. Optimizing Real-Time performance: I would want to optimize my code to allow more reaction time for the robot such that it could handle minor error in position estimation made by the AI model. This may include using more FPGA core for processing as they are faster.

3. Hybrid Motion Prediction Models: To better handle complex motion dynamics of juggling, I will explore physics-informed machine learning model as it could reduce the workload and improve accuracy of the prediction.

Appendix

I have the need to use the kria platform in headless mode as I do not have a DP-enabled monitor with me. Hence, I have configured the KR260 running Ubuntu to run headless VNC where my laptop connected to the same network could access the desktop. The virtual display can be activated by running the following script obtained from this answer.