Hackster is hosting Hackster Holidays, Ep. 7: Livestream & Giveaway Drawing. Watch previous episodes or stream live on Friday!Stream Hackster Holidays, Ep. 7 on Friday!
samarth
Created July 31, 2024

Autonomous Self-Navigating AI Vision-Guided Robot Using AMD

This project addresses the critical need for autonomous navigation in challenging environments, where traditional navigation methods are inf

8
Autonomous Self-Navigating AI Vision-Guided Robot Using AMD

Things used in this project

Hardware components

Kria™ KR260 Robotics Starter Kit
AMD Kria™ KR260 Robotics Starter Kit
×1
Pmod HB3
Digilent Pmod HB3
×2
DC Motor, 12 V
DC Motor, 12 V
×2
Webcam, Logitech® HD Pro
Webcam, Logitech® HD Pro
×1
AA Batteries
AA Batteries
×2

Software apps and online services

Vitis Unified Software Platform
AMD Vitis Unified Software Platform

Story

Read more

Schematics

Schematic

This is the schematic of all the connections to the kria

Code

The Flask Server running on the Kria

Python
This is the main code that keeps running on the Kria, we can control the robot by hitting these endpoints
from flask import Flask, send_from_directory, render_template, jsonify
import cv2
import os
from gradio_client import Client
import time
app = Flask(__name__)

#============CAMERA PART===============================#

# Directory to save images
IMAGE_DIR = './main/static/images'

# Ensure the directory exists
if not os.path.exists(IMAGE_DIR):
    os.makedirs(IMAGE_DIR)

def capture_image():
    # Initialize the camera (use  0 for the default camera)
    cap = cv2.VideoCapture(3)

    # Check if the camera is opened correctly
    if not cap.isOpened():
        raise IOError("Cannot open camera")

    # Capture a single frame
    ret, frame = cap.read()

    # Release the camera
    cap.release()

    if ret:
        # Save the image
        path = f"image_{int(time.time())}.jpg"
        filename = os.path.join(IMAGE_DIR, path)
        cv2.imwrite(filename, frame)
        print(f"Image saved to {filename}")
    else:
        print("Failed to capture image")

    return path


def process_image(path):
    client = Client(
        "https://ybelkada-llava-1-5-dlai.hf.space/--replicas/f3edg/")
    result = client.predict(
        "Describe this image",  # str  in 'parameter_0' Textbox component
        path,  # filepath  in 'parameter_1' Image component
        api_name="/predict"
    )
    print(result)
    return result

@app.route('/capture')
def capture_and_save():
    path = capture_image()
    path = './main/static/images/'+path
    print(path)
    desc = process_image(path)
    # return desc
    return jsonify({'image_url': path, 'description': desc})

@app.route('/')
def home():
    return render_template('index.html')

# @app.route('/get_car_data')
# def get_car_data():
#     # Example data
#     image_url = 'static/images/image_1708585084.jpg'
#     description = 'This is a description of the car.'
    
#     # Return the data as JSON
#     return jsonify({'image_url': image_url, 'description': description})

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0')

Credits

samarth
1 project • 1 follower
I like to build stuff

Comments