Grocery Store Object Recognition on Kria KR260

AI-Based Grocery Store Display Enhancement

Inspiration

In today's fast-paced world, the visual appeal of food plays a crucial role in customer satisfaction and dining pleasure. We were inspired to create an innovative solution that enhances the visual appeal of food using AI technology, ultimately encouraging healthier food choices in grocery stores.

Team Members

Our team is composed of five male university students from Korea:

• Team Leader: Ki-Jeong Kong
• Team Members: Young-Ki Kim, Gun-Hyung Park, Seung-Min Baek, and Jin-Il Lee

What it does

Our project implements an object recognition system for grocery store products using the Kria KR260 Robotics Starter Kit. The system recognizes different food items and is designed to adjust lighting to enhance their appearance, making fresh produce, meats, and dairy products more visually appealing to shoppers.

Hardware Used

For this project, we utilized the Kria™ KR260 Robotics Starter Kit. This advanced development kit, based on Xilinx's Kria System on Module (SoM), was chosen for its powerful features that align perfectly with our project requirements:

• Kria K26 SoM: At the heart of the kit is a high-performance FPGA-based module that includes programmable logic and an ARM Cortex-A53 processor. This combination allows for flexible implementation of both hardware and software logic.
• Expandability: The kit provides various interfaces and expansion ports, supporting a wide range of robotics and automation applications. This feature was crucial for our plans to integrate additional components like LED lighting systems.
• Ease of Development: The kit is compatible with the PYNQ (Python Productivity for Zynq) framework, enabling rapid prototyping and development. This compatibility significantly accelerated our development process.
• AI and Vision Capabilities: The KR260 offers examples and reference designs for AI processing and computer vision functionalities, which were essential for our object recognition system.

We chose the Kria KR260 Robotics Starter Kit because our project required a high-performance FPGA to handle AI computations, along with an ARM-based processor to implement both hardware and software logic. This kit provided the ideal balance of processing power, flexibility, and ease of use for our AI-based grocery store display enhancement system.

How we built it

We developed our project using the Kria KR260 Robotics Starter Kit, leveraging its powerful AI capabilities. Our development process included:

• Data Preparation: We used the "Grocery Store Dataset" from Kaggle, organizing it using a custom notebook (prepare_grocery_dataset.ipynb).
• Transfer Learning: We fine-tuned a ResNet50 model for our specific use case using TensorFlow 2.x.
• Model Inspection: We verified the compatibility of each layer with the Vitis AI DPU IP using a custom notebook (inspect_resnet50_grocery.ipynb).
• Model Quantization: We quantized the fine-tuned model for improved efficiency on the KR260 using Vitis AI-TensorFlow 2
• Compilation: We compiled the quantized model specifically for the Kria KR260 board using a custom script (compile_resnet50_grocery.sh) and the KR260's DPU architecture fingerprint.
• Deployment: We successfully ran the object recognition model on the KR260 board, integrating it with a USB webcam for real-time object recognition.

Project Structure

Our project follows a well-organized structure:

.
├── compile_resnet50_grocery.sh
├── compiled_model/
├── float_model/
├── grocery_store_dataset/
├── inspect_resnet50_grocery.ipynb
├── kr260.json
├── logs/
├── prepare_grocery_dataset.ipynb
├── quant_model/
├── quantize_resnet50_grocery.ipynb
├── run_resnet50_grocery_kr260.ipynb
├── train_resnet50_grocery.ipynb
├── train_resnet50_grocery_colab.ipynb
└── webcam/

This structure ensures easy navigation and management of different aspects of the project.

Challenges we ran into

Our project faced several significant challenges, primarily related to hardware issues and time constraints:

1. Boot Firmware Issues: We encountered issues similar to those reported for the KV260, likely related to outdated boot firmware

• The board would occasionally lock up during soft boot.
• We observed LED patterns indicating that the SOM had powered down unexpectedly: CC PL, SOM PL, CC PS, and PWR IN LEDs were ON, while SOM FPD and SOM LPD LEDs were OFF.

• This issue likely stemmed from problems with the PS_PWR_EN signal control during Linux reboot, as outlined in Xilinx Answer Record 71952.
• Boot Firmware Issues: We encountered issues similar to those reported for the KV260, likely related to outdated boot firmware:

2. Hardware Malfunction: Our initial Kria KR260 board malfunctioned:

• The system failed to boot into Ubuntu, and no text was displayed on the UART connected via micro-USB.
• Using Vivado, we received a critical warning: [Labtools 27-3421] xczu5_0 PL Power Status OFF, cannot connect PL TAP. Check POR_B signal.

• The register values indicated the boot mode was incorrectly set to PS JTAG mode.
• Replacement and Delays: After identifying the hardware issue, we had to request after-sales service and eventually purchase a new board from AMD's official website. This process took about a month, significantly delaying our progress.
• Time Constraints: Due to the delays, we had to adjust our project scope:
• We switched from our initial plan of using YOLO for object detection to using a pretrained ResNet50 model.
• We successfully implemented the object recognition system but couldn't complete the LED lighting integration before the competition deadline.

Boot Firmware Update

To address the boot firmware issues, we recommend the following steps:

• Ensure your Kria board's boot firmware is up to date before beginning development.
• Follow the instructions on the Xilinx Wiki page for Kria SOMs & Starter Kits: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1641152513/Kria+SOMs+Starter+Kits#Boot-FW-update-with-xmutil
• Use the xmutil command to perform the boot firmware update as detailed in the guide.

This simple step can prevent many issues and save significant time and resources.

Accomplishments

• Successfully implemented a robust object recognition system on the Kria KR260.
• Achieved impressive processing times:
  Pre-processing time: ~4.5 ms
  DPU execution time: ~17.2 ms
  Post-processing time: ~0.34 ms
• Created a well-structured, reproducible project with detailed documentation and notebooks for each step of the process.

Learned

This project taught us valuable lessons in:

• AI model optimization for edge devices, including quantization and compilation for specific hardware.
• Working with the Kria KR260 platform and its Deep Learning Processing Unit (DPU).
• The importance of proper firmware management in embedded systems development.
• Insights into potential applications of AI in enhancing retail experiences.

What's next for AI-Based Grocery Store Display Enhancement

As today marks the final submission deadline for the competition, we want to acknowledge that we weren't able to fully complete all aspects of our original vision. However, we are committed to continuing this project beyond the competition, driven by our passion for innovation and the potential impact of our system. Our future plans include:

1. Custom DPU IP with GPIO:

• Build a custom Deep Learning Processing Unit (DPU) IP that includes General-Purpose Input/Output (GPIO) functionality.
• This will facilitate smoother interaction between hardware and software, enabling communication with various external devices.
• Particularly useful for setting up a real-time feedback system in a supermarket environment.

2. YOLO Implementation in PetaLinux:

• Implement the latest version of YOLO (You Only Look Once) object detection algorithm in a PetaLinux environment.
• Utilize real-time object recognition to track customer movements and optimize product placement.
• Enhance shopping experiences and enable more efficient store management.

3. Visual Feedback System:

• Integrate 3-color LEDs connected via GPIO to provide visual feedback to customers.
• Use colors to convey information about products, creating stronger impressions and enhancing the overall shopping experience.

4. LED Lighting Integration:

• Develop LED control software that responds to object recognition results.
• Create adaptive lighting scenarios based on recognized products and customer behavior.

5. User Interface Development:

• Design and implement a comprehensive user interface for system management and configuration.
• Include features for real-time monitoring and adjustment of the visual enhancement system.

6. Simulated Environment Testing:

• Conduct thorough testing in simulated grocery store environments to validate system performance and user experience.

7. Dataset Expansion:

• Expand our training dataset to include a wider variety of grocery items, improving recognition accuracy and versatility.

8.Performance Optimization:

• Continuously refine and optimize our models and algorithms for even faster inference times and better energy efficiency.

While we couldn't fully realize our initial vision within the competition timeframe, this experience has deepened our understanding of hardware-software integration in edge AI applications. We're excited to continue working on this project, aiming to improve customer experiences and contribute to the field of AI-enhanced retail environments.

References

• "Grocery Store Dataset" from Kaggle: https://www.kaggle.com/datasets/validmodel/grocery-store-dataset/data
• Vitis-AI Tutorials repository: https://github.com/Xilinx/Vitis-AI-Tutorials/tree/3.5/Tutorials/RESNET18/
• Xilinx Answer Record 71952 for boot firmware issues.
• Xilinx Wiki page for Kria SOMs & Starter Kits: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1641152513/Kria+SOMs+Starter+Kits#Boot-FW-update-with-xmutil