Lego Land Rover Defender Robot with Kria KR260

What happens when you stumble across a Lego kit of your dream car and you also just received a shiny, new Kria KR260 Robotics Starter Kit? Well that's a robot project just begging to be built...

About 10 years ago during my senior year of my undergraduate degree in engineering, I was tasked with building a robot capable of autonomously navigating a variable obstacle course (meaning no hard-coding any drive routes) and identifying a given object to pick up as my senior design project for the year. At this point in time, the Raspberry Pi 2 had just come out and Arduinos were the popular kids on the block.

Ultimately, what was born was a rover 5 chassis driven by a trio of Arduinos with a few ping sensors for detecting the obstacles in the course, an array of IR emitter and receiver LEDs to detect block shapes (since the shapes were black on a white background, that was easier than getting a camera up and running at the time), and an OWI Inc Robotic Arm for grabbing the selected block shape.

So when the Land Rover Defender Lego Technic kit "magically appeared" on my doorstep and AMD-Xilinx asked me if I had any ideas of what to do with one of their Kria KR260 Robotics Starter Kits, my old senior design robot was the first thought that came to mind. I though that'd be fun to see how I could recreate a similar version of it now that I'm 10 years into my career and have all of these cool new tools at my disposal.

Lego Build

First things first, the upgrade from the rover 5 chassis to the Land Rover Defender Technic meant playing with Legos again for the first time in my adult life. I don't know how I hadn't discovered the world of Lego Technic Kits prior to Facebook ads showing me the Defender kit, but it's probably going to be to the detriment of my wallet in the future...

While I was super excited about putting on a good podcast and putting together this kit, I immediately realized that this would take a little more strategy than blindly following the build instructions for a relaxing Sunday afternoon.

When I opened the much-larger-than-expected box that showed up on my doorstep, the 20 separate baggies of parts and 500-page/1000-step build guide was a bit daunting. The Defender was obviously more complex than just a chassis with 4 wheels connected to a common drive train.

I took a step back to do some research to find that the Defender kit had a full on mechanic motor, with 4-gear transmission, and front/rear differentials. This meant that the Defender had switches to go between the 4 gears, switch between forward/neutral/reverse, and switch between high and low gear.

It was immediately obvious that my original "simplistic" plan of one motor to drive the wheels and a second motor to steer the Defender could easily snowball into a massively complex mechanical setup of switches and whatever else to be able to give the Kria complete operational control of switching gears.

Bonus challenge: the gearbox was controlled by a small wheel meant for a thumb to turn and was placed such that I'd have to completely redesign the Lego build to access it with any other hardware.

This was the final straw that made me decide to leave the gearbox set in low and the transmission set to neutral so I could return to my original "simplistic" plan and change between forward/reverse by changing the polarity of the supply voltage to the single motor connected to the drive train instead. I'm sure the mechanical challenge of coming up with switching mechanisms to give the Kria control of all these extra drive features sounds fun to some, but it's not my cup of tea...

I found a couple other users that had talked about the challenges they encountered motorizing the Defender, both talked about needing to completely redesign/rebuild the gearbox to be able to use the Lego motor kits. This wasn't exactly something I was keen on doing because I was more anxious to get into the FPGA design on the Kria and not wind up in a black hole of Lego redesign. So I decided that I'd need to find points in the drive train that always turned/moved regardless of direction the wheels were turning with the transmission in neutral and the gearbox in 1 (out of 4) low, then I'd just have to figure out how to attach a regular hobby motor to one of those points. These are the point circled in pink in the photo above.

The steering motor thankfully appeared to not have any hidden surprises in terms of complexity. The Lego design had it simply coming up to a gear through the roof which made it easily accessible to connect a motor with encoder to it.

After looking up the final dimensions of the Lego Defender, I realized it just so happened that the Kria KR260 carrier board was the exact perfect size to be an "aftermarket" roof rack.

The build guide was broken up into four overall stages (with each part baggie labeled accordingly) so I decided the best course of action was to stop after each stage to reevaluate the motor placement/configuration for driving and steering motors.

Controlling Motors with the KR260

The KR260 carrier board for the Kria SoM was designed with robotics applications in mind, so there is plenty of I/O for driving common robotics peripherals such as motors. Four Pmod 12-pin connectors with 8 I/Os each and a Raspberry Pi HAT header with 26 I/Os are the ideal options for driving I/Os like hobby motors.

But before I could start using the Pmods or Raspberry Pi HAT header, I needed to map out each pin from the physical connector up through to its respective I/O number in the software to target. Since the Pmods and Raspberry Pi HAT header are connected to the Kria's programmable logic (PL), this meant I needed some sort of HDL design that instantiated an GPIO interface connected to the Zynq MPSoC ARM-core processors so the application space could access the pins.

I knew that I wanted to use the Ubuntu 22.04 distribution made for the Kria (see installation instructions here) since I wanted to install Edge Impulse for the autonomous obstacle avoidance element of the project. And since I was needing to program the PL with a different bitstream with the GPIO interface design, I figured the best course of action was to load the new bitstream and a corresponding device tree overlay with the Xilinx Platform Management Utility (xmutil) in the same method as if I were going to run an accelerated application on it. This meant I didn't have to make any modifications to the Ubuntu image itself or try to rebuild it in any way.

The Ubuntu distribution made for the Kria has xmutil built into it so I was able to reuse my bitstream and device tree overlay from my previous project post where I connected to the Pmods or Raspberry Pi HAT header using AXI GPIO IP blocks. While I used PetaLinux in that post, all of the steps can be followed exactly the same for Ubuntu 22.04 on the KR260 (just skip the step of Update PetaLinux with New XSA).

I simply transferred the existing bitstream, device tree blob, and shell.json files I had from that project to my KR260 running Ubuntu 22.04:

~$ scp kr260_gpio.dtbo kr260_gpio.bit.bin shell.json petalinux@<KR260’s IP address>:/home/ubuntu/kr260_gpio

Since I'm treating it like an accelerated application, a directory with the common name of the PL design files needs to be created in /lib/firmware/xilinx and the PL design files copied/moved into it so that the Kria' s resource manager daemon is aware of the files and can access them with the appropriate permissions via the xmutil command:

ubuntu@kria:~$ sudo mkdir /lib/firmware/xilinx/kr260_gpio
ubuntu@kria:~$ sudo cp ./kr260_gpio/* /lib/firmware/xilinx/kr260_gpio/

And I was able to load it the exact same way:

ubuntu@kria:~$ sudo xmutil listapps
ubuntu@kria:~$ sudo xmutil unloadapp
ubuntu@kria:~$ sudo xmutil loadapp kr260_gpio

Upon loading the new bitstream and device tree overlay, the KR260 prints out the confirmation of the new hardware it sees, which now includes the Pmods and Raspberry Pi header:

[ 2018.574600] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/firmware-name
[ 2018.584724] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/resets
[ 2018.594824] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/overlay0
[ 2018.604688] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/overlay1
[ 2018.614558] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/afi0
[ 2018.624055] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/clocking0
[ 2018.633984] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/clocking1
[ 2018.643913] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/overlay2
[ 2018.653758] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_intc_0
[ 2018.663775] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/misc_clk_0
[ 2018.673792] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/pmod_1
[ 2018.683505] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/pmod_2
[ 2018.693199] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/pmod_3
[ 2018.702894] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/pmod_4
[ 2018.712567] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/rpi_gpio
kr260_gpio: loaded to slot 0
[ 2018.928870] zocl-drm axi:zyxclmm_drm: IRQ index 32 not found

This gave me access to the Pmods using the Sysfs driver in Ubuntu so I could write a script in any desired language to control Digilent HB3 Pmod H-bridge drivers with encoder feedback inputs.

To find out which GPIO bank number equated to which Pmod, simply run the list (ls) command with -la flag on the /sys/class/gpio directory and the addresses of the AXI GPIOs will correlate back to those set in the Vivado block design address editor:

I did find that there were some odd permissions issues with the Sysfs driver in Ubuntu that necessitated me manually exporting the each of the GPIO pins (6 in total) and changing the owner of the new directories back to the ubuntu user:

ubuntu@kria:~$ sudo chown ubuntu:ubuntu -R /sys/class/gpio/*
ubuntu@kria:~$ echo 310 > /sys/class/gpio/export
ubuntu@kria:~$ sudo chown ubuntu:ubuntu -R /sys/class/gpio/gpio310/*

I was then able to execute the following commands either from the command line or a script as normal:

ubuntu@kria:~$ echo out > /sys/class/gpio/gpio310/direction
ubuntu@kria:~$ echo 1 > /sys/class/gpio/gpio310/value
ubuntu@kria:~$ echo 0 > /sys/class/gpio/gpio310/value

Just to double-check my pin mappings, I used some LEDs and did some toggle tests from the Kria's command line:

With the Pmod pin mappings verified, I was able to hook up the N20 motors with encoders I selected for the drive train and steering gear and write the simple Python script to control the motor's direction and read the encoder values:

import os
import sys
import time
import subprocess
import multiprocessing

while True:
    os.system('echo 1 > /sys/class/gpio/gpio318/value') # set direction to forward
    os.system('echo 1 > /sys/class/gpio/gpio319/value') # enable motor

    # read the encoder values 
    for x in range(10):
        encoderApin = open('/sys/class/gpio/gpio320/value', 'r')
        encoderA = encoderApin.read()
        time.sleep(0.1)

        encoderBpin = open('/sys/class/gpio/gpio321/value', 'r')
        encoderB = encoderBpin.read()
        time.sleep(0.1)
        print('encoderA = ', encoderA, ' encoderB = ', encoderB)

    os.system('echo 0 > /sys/class/gpio/gpio319/value')
    os.system('echo 0 > /sys/class/gpio/gpio318/value')

A couple drawbacks here: every time the Kria is power cycled, the bitstream with device tree overlay for the AXI GPIO has to be reloaded. Then I also have to manually export and change owner of each of the GPIO's directories back to the ubuntu user. I'm sure there is a way to fix this and modify the startup script to load my custom PL design instead of the default accelerated application, but that's all a problem I can solve in revision 2 of this robot... I'm just anxious to get things actually moving for now.

Drive Train Motor Setup

With the basics of motor control handled from the Kria's side, the next step was to get the motors mounted and hooked up starting with the motor for the drive train of the Defender.

I ultimately found that the driveshaft running from the front wheels to the back wheels was the easiest point of access that satisfied my requirement of being a point in the drive train that always turned/moved regardless of direction the wheels were turning with the transmission in neutral and the gearbox in 1 (out of 4) low.

I simply had to spin the driveshaft rod in one direction for the Defender to roll forward and the opposite for reverse, so perfect! As far as physically connecting the N20 motor goes, I picked up some gear wheels and belt for an Enders Pro 3D printer X and Y-Axis belt pulley.

There was a perfectly spaced gap to access the driveshaft from the drivers seat of the Defender so I mounted the N20 motor in the driver's seat with one Enders Pro gear attached to it and a placed the other Enders Pro gear on the driveshaft itself. Then I simply ran a belt pulley between the two.

This is where I hit a major snag: the little 6V N20 motor could spin the wheels when the body of the Defender was suspended in air, but when the Defender was placed on the ground its own weight was too much and the N20 couldn't budge it.

But that wasn't my only issue... To my horror, when the N20 was able to turn the wheels (with the Defender was suspended in air) I also noticed the entire frame of the Defender was flexing due to the required tension on the pulley belt. The Legos themselves just weren't as rigid enough for this application. So while it wasn't a total failure/no-go, the setup was prone to the belt slipping every so often (making a terrible ticking noise as the teeth slipped against the gears).

This led to me purchasing the Technic spare parts kit off Amazon to create beefier mounts for both the motors and KR260 itself and make it such that everything could easily be replaced given the break factor seemed fairly high at this point from a mechanical perspective. Permanently mounting anything via methods like super glue, etc. seemed like I'd probably just be shooting myself in the foot later on.

Along with beefing up and reinforcing the motor mount itself, I also needed a bigger motor. So I was off to my local MicroCenter to wander the aisles until I found a solution. Thankfully, I must have had some good karma built up because I found a lone OSEPP high torque electric motor on the shelf that was also 6V so I could easily drop in in place of the N20.

I did end up having to completely take out the driver's seat to build in the mount for the new OSEPP motor since it was quite a bit bigger than the N20, but it ended up being perfect and I was able to build some extra support for the frame by taking advantage of the driver's side footwell.

Joining the pulley belt together still caused a stiff spot in the belt that creates extra tension when it passes around a gear and a tooth slip or two, so there was still an annoying tick when the Defender drove forward. But it was ultimately a much more stable/sound setup.

KR260 Roof Rack

Removing the drivers seat ended up being a good thing in more way than one because some of the pieces for the drivers seat ended up being exactly what I needed to build the roof rack to hold the KR260 in place:

Since the gear for the steering column was also on the drivers side, I oriented the KR260 such that the Pmod connectors with the HB3 boards could be wired up to the N20 on the roof for the steering column and the OSEPP motor in the place of the driver's seat.

I created brackets out of Legos around the edges of the Defender's roof rack to hold the KR260 board in place and keep it from sliding off the roof of the Defender, and I could simply lift it up to take it off the Defender when needed.

Steering Motor Setup

The mount for the N20 motor for steering ended up not looking as graceful as I would have liked, but it got the job done. Holding the motor parallel with the steering rod in the roof and sacrificing one Lego piece (the red piece in the photo below) to get it to fit properly on the N20 ended up being the most mechanically sound option with the parts I had at my disposal:

And with that, it was finally time to start on the main aspect of the project: making this thing capable of some form of autonomous driving.

Overall, with the upgrade to the OSEPP high torque motor and the final setup of the N20 steer motor, I ended up with the final over pinout for the KR260's Pmods with the HB3. This is mapped all the way from Kria FPGA package pin to the final Linux Sysfs GPIO number:

And to supply the power to the motors the 6V they needed, I used 2.1mm female barrel jack cable pigtails to connect to the M+/M- screw terminals of the HB3 Pmod boards connected to a 6V 2.1mm barrel jack wall adapter using a female to male 2.1mm splitter cord.

Obstacle Avoidance ML Model

While I'm skipping the arm to pick up shaped blocks in this remake of my senior design robot, the autonomous obstacle avoidance was the element I really wanted to focus on remaking since really cool tools like Edge Impulse now exist. This ultimately led to be describing the project to non-technical friends and family as a "glorified Roomba".

Edge Impulse is capable of creating some pretty complex models, but I wanted to start with the simplest version of obstacle avoidance and iterate from there. And what's the simplest form of obstacle avoidance you might ask? Pick an object and stop if you see it.

Some explanation on this: to train an image processing type ML model, you have to tell it what to look for. And the more general/generic that thing is, the more complex the ML model itself and training it becomes. This is what led me to decide on picking one specific object as my "obstacle" to avoid to train my ML model in Edge Impulse to look for.

My cute little 3D printed jack-o-lantern from this past Halloween ended up being my "obstacle". It's been sitting on my desk staring at me waiting for a task for too long now.

So I connected my KR260 to Edge Impulse Studio and started capturing test data identifying my little jack-o-lantern to create my ML model with. See my previous post for the instructions on how to install Edge Impulse on the KR260 and use it in Edge Impulse Studio here.

I used my nice Logitech USB webcam since I know it's compatible with all of the drivers on the KR260 Linux images. I originally intended on mounting it to the hood of the Defender, but I found that it was way too heavy so I just have it placed off to the side for the moment.

I learned the hard way in my past project where I created a digital menu for my local coffee shop with a live webcam feed of a pastry case to allow customers to see what pastries were still available from the drive-thru, that not all webcams play nice with embedded Linux. But once the core code is working, it's just a matter of plugging in different smaller ones until I find one that works so I didn't think that was worth holding up the project with at this point.

Going back to Edge Impulse Studio, I choose to have the model labeling method for the ML model to be bounding boxes since part of the bounding box info output from the Edge Impulse Runner service is where the bounding box is located in the frame of the image. Even though my first iteration of the drive code on the Defender would just be to stop if the ML model saw the jack-o-lantern anywhere in the frame, I'd need to know where in the frame it was to be able to add the code for steering around it later.

After successfully training my ML model to look for my little jack-o-lantern, I downloaded the.eim file for it to the Kria for it to use locally.

I created a directory with the same name as my project in Edge Impulse Studio to download the.eim file into just to keep things organized on my KR260:

ubuntu@kria:~$ mkdir -p kr260-pumkpin-obstacle
ubuntu@kria:~$ cd ./kr260-pumkpin-obstacle
ubuntu@kria:~/kr260-pumkpin-obstacle$ sudo edge-impulse-linux-runner --download modelfile.eim

In my previous post detailing how to install and use Edge Impulse on the KR260, I cloned the Edge Impulse Linux Python SDK onto my Kria KR260 since that was the language I chose to write the drive control in.

Like I previously mentioned, there are example scripts in that SDK that demonstrate how to use/call the Edge Impulse Runner service for each data type, so I copied those into the directory with my ML model.eim file:

ubuntu@kria:~/kr260-pumkpin-obstacle$ cp ../linux-sdk-python/examples/image/classify-image.py ./
ubuntu@kria:~/kr260-pumkpin-obstacle$ cp ../linux-sdk-python/examples/image/classify.py ./

One of the Python examples (classify.py) takes the directory location of the ML model.eim file and target video source (camera) index as inputs then continuously capture images from that camera and passes them through the ML model for classification. It then just prints the results of the classification of the image to the serial console. And it will forever do this until the user kills the script with ctrl+C.

For example, since I trained my ML model to look for my little jack-o-lantern and place a bounding box around it, the classify.py script will print out if it saw the jack-o-lantern and the corresponding info for the bounding box it placed around it in the image (which includes the size and location of the bounding box in the image).

So as a starting point, I just ran the example classify.py script as-in to make sure everything was working so far:

ubuntu@kria:~/kr260-pumpkin-obstacle$ python3 classify.py /home/ubuntu/kr260-pumpkin-obstacle/modelfile.eim 0

Placing the jack-o-lantern in the view of the Logictech webcam gave me the following successful result:

To break down this output a little more: the script is printing out how many bounding boxes the ML model placed on the image (i.e. - the number of target objects it sees) and the x/y/w/h data of the bounding box(es). Where:

• x = x-axis coordinate in the image of the upper left-hand corner of the bounding box
• y = y-axis coordinate in the image of the upper left-hand corner of the bounding box
• w = width dimension of the bounding box
• h = height dimension of the bounding box

This script was the perfect starting point for this project because it did a majority of the leg work for me with getting the Edge Impulse Runner service working with my webcam and running images through the ML model. All's I had to do was find the point in the code where it was printing the bounding box info and replace it with my code for controlling the motors for driving and steering.

As a quick side note, this is essentially what the entire logic of dealing with ML models in code can boiled down to: passing source data (images in this case) through the model and then making decisions on what to do based on its output.

So lines 105 - 126 in the original code of classify.py are where I focused:

.
.
.
if "classification" in res["result"].keys():
                print('Result (%d ms.) ' % (res['timing']['dsp'] + res['timing']['classification']), end='')
                for label in labels:
                    score = res['result']['classification'][label]
                    print('%s: %.2f\t' % (label, score), end='')
                print('', flush=True)

            elif "bounding_boxes" in res["result"].keys():
                print('Found %d bounding boxes (%d ms.)' % (len(res["result"]["bounding_boxes"]), res['timing']['dsp'] + res['timing']['classification']))
                for bb in res["result"]["bounding_boxes"]:
                    print('\t%s (%.2f): x=%d y=%d w=%d h=%d' % (bb['label'], bb['value'], bb['x'], bb['y'], bb['width'], bb['height']))
                    img = cv2.rectangle(img, (bb['x'], bb['y']), (bb['x'] + bb['width'], bb['y'] + bb['height']), (255, 0, 0), 1)
                
             if (show_camera):
                cv2.imshow('edgeimpulse', cv2.cvtColor(img, cv2.COLOR_RGB2BGR))
                if cv2.waitKey(1) == ord('q'):
                    break

            next_frame = now() + 100

    finally:
        if (runner):
        runner.stop()
.
.
.

And I ended up with the modified code below for the simple drive forward and stop if jack-o-lantern is seen. Starting with enabling the drive motor right before processing the bounding box output so it'd stay running if no jack-o-lantern is seen, then only stopping when a bounding box with the label "pumpkin" is found ("pumpkin" is the label I used when creating/training the model in Edge Impulse Studio). This motor would also be enabled again as soon as bounding boxes labeled "pumpkin" where no longer being output from the model.

.
.
.
if "classification" in res["result"].keys():
                print('Result (%d ms.) ' % (res['timing']['dsp'] + res['timing']['classification']), end='')
                for label in labels:
                    score = res['result']['classification'][label]
                    print('%s: %.2f\t' % (label, score), end='')
                print('', flush=True)

            elif "bounding_boxes" in res["result"].keys():
                print('Found %d bounding boxes (%d ms.)' % (len(res["result"]["bounding_boxes"]), res['timing']['dsp'] + res['timing']['classification']))

                if (len(res["result"]["bounding_boxes"])) == 0:
                    os.system('echo 1 > /sys/class/gpio/gpio311/value')

                for bb in res["result"]["bounding_boxes"]:
                    print('\t%s (%.2f): x=%d y=%d w=%d h=%d' % (bb['label'], bb['value'], bb['x'], bb['y'], bb['width'], bb['height']))
                    img = cv2.rectangle(img, (bb['x'], bb['y']), (bb['x'] + bb['width'], bb['y'] + bb['height']), (255, 0, 0), 1)
                    if (bb['label']) == "pumpkin":
                        os.system('echo 0 > /sys/class/gpio/gpio311/value')
                    else:
                        os.system('echo 1 > /sys/class/gpio/gpio311/value')
                
             if (show_camera):
                cv2.imshow('edgeimpulse', cv2.cvtColor(img, cv2.COLOR_RGB2BGR))
                if cv2.waitKey(1) == ord('q'):
                    break

            next_frame = now() + 100

    finally:
        if (runner):
        runner.stop() 
        # make sure motor is off when exiting the script
        os.system('echo 0 > /sys/class/gpio/gpio311/value')
.
.
.

That last line is pretty important for making sure the motor is stopped when the script is killed from the serial console. That way the Defender didn't end up in "runaway" mode while I was debugging (because that totally didn't happen.... again....).

Which is why I have my make shift "floor jack" for the Defender for testing the drive motor code now.

When I said that this "happened again" I'm referring to when I learned this lesson the hard way with my senior design robot. The logic in my code didn't work like I thought it did and the robot took off dragging my laptop behind it (cut to that 8 second clip).

Also, just to reiterate, the GPIO pin 311 for enabling the motor on the HB3 Pmod in the Pmod 1 slot of the KR260 was already exported and set as an output pin manually before executing the above code in the modified classify.py script.

ML Model Control of Motor Driving

With an basic proof-of-concept working of my obstacle avoiding Lego Defender, I moved on to the next iteration of my obstacle avoidance code: deciding how to steer around the jack-o-lantern when it appears in the image.

Since I know exactly where the bounding box is located the image based on the data the Edge Impulse Runner service gives me, I updated the Python script to check the x-axis value of the bounding box (which is where the bounding box is located on the x-axis of the image) then steer left if the bounding box is located in the right half of the image and steer right if the bounding box is located in the left half of the image.

To determine the exact size of the image in terms of the x values being output from the Edge Impulse Runner service, I placed the webcam and jack-o-lantern on the same flat surface and slid the jack-o-lantern left and right respectively while running the original classify.py script and made note of the values for x the script output right before I slid the jack-o-lantern far enough to the left/right that it was no longer in frame.

I found that the left-most edge of the image was x-axis value 0 (x = 0) the right-most edge of the image was x-axis value 90 (x = 90). And given that the x-axis value is the upper left-hand corner of the bounding box, I made the determination that any bounding box with an x value greater than or equal to 40 was located in the right half of the image and any bounding box with an x value less than 40 was located in the left half of the image.

So I updated my Python code at the point after it's found a bounding box with the label "pumpkin" to then check the x value of that bounding box and steer left/right accordingly, then returning the wheels to a straight direction again:

.
.
.
if "classification" in res["result"].keys():
                print('Result (%d ms.) ' % (res['timing']['dsp'] + res['timing']['classification']), end='')
                for label in labels:
                    score = res['result']['classification'][label]
                    print('%s: %.2f\t' % (label, score), end='')
                print('', flush=True)

            elif "bounding_boxes" in res["result"].keys():
                print('Found %d bounding boxes (%d ms.)' % (len(res["result"]["bounding_boxes"]), res['timing']['dsp'] + res['timing']['classification']))

                if (len(res["result"]["bounding_boxes"])) == 0:
                    os.system('echo 1 > /sys/class/gpio/gpio311/value')

                for bb in res["result"]["bounding_boxes"]:
                    print('\t%s (%.2f): x=%d y=%d w=%d h=%d' % (bb['label'], bb['value'], bb['x'], bb['y'], bb['width'], bb['height']))
                    img = cv2.rectangle(img, (bb['x'], bb['y']), (bb['x'] + bb['width'], bb['y'] + bb['height']), (255, 0, 0), 1)
                    if (bb['label']) == "pumpkin":
                        if (bb['x'] >= 40):
                            if (defender_dir == straight):
                                defender_dir = half_turn_left(defender_dir)
                            else: 
                                defender_dir = full_turn_left(defender_dir)
                    else:
                        os.system('echo 1 > /sys/class/gpio/gpio311/value')
                
             if (show_camera):
                cv2.imshow('edgeimpulse', cv2.cvtColor(img, cv2.COLOR_RGB2BGR))
                if cv2.waitKey(1) == ord('q'):
                    break

            next_frame = now() + 100

    finally:
        if (runner):
        runner.stop() 
        # make sure motor is off when exiting the script
        os.system('echo 0 > /sys/class/gpio/gpio311/value')
.
.
.

This is where I discovered that the N20 motor's encoder didn't give me fine enough resolution for determining how far I had turned the wheels, which is pretty important because I can't try to turn the wheels with the steering column too far before I break something.

So I had to backtrack and write some logic to keep track of the current wheel direction and turn it in timed half-turn increments:

left = 1
straight = 0
right = 2

def half_turn_left(wheel_dir_current):
    if (wheel_dir_current == straight):
        wheel_dir = left
        os.system('echo 1 > /sys/class/gpio/gpio318/value') # turn left
        time.sleep(0.1)
        os.system('echo 1 > /sys/class/gpio/gpio319/value')
        time.sleep(0.3)
        os.system('echo 0 > /sys/class/gpio/gpio319/value')
        time.sleep(0.1)
    elif (wheel_dir_current == right):
        wheel_dir = straight
        os.system('echo 1 > /sys/class/gpio/gpio318/value') # turn left
        time.sleep(0.1)
        os.system('echo 1 > /sys/class/gpio/gpio319/value')
        time.sleep(0.3)
        os.system('echo 0 > /sys/class/gpio/gpio319/value')
        time.sleep(0.1)
    else:
        wheel_dir = wheel_dir_current
        print('Wheels cannot turn any further left...')

    return wheel_dir

def full_turn_left(wheel_dir_current):
    if (wheel_dir_current == right):
        wheel_dir = left
        os.system('echo 1 > /sys/class/gpio/gpio318/value') # turn left
        time.sleep(0.1)
        os.system('echo 1 > /sys/class/gpio/gpio319/value')
        time.sleep(0.6)
        os.system('echo 0 > /sys/class/gpio/gpio319/value')
        time.sleep(0.1)
    else:
        wheel_dir = wheel_dir_current
        print('Wheels cannot make full turn left, try half turn left...')

    return wheel_dir

def half_turn_right(wheel_dir_current):
    if (wheel_dir_current == straight):
        wheel_dir = right
        os.system('echo 0 > /sys/class/gpio/gpio318/value') # turn right
        time.sleep(0.1)
        os.system('echo 1 > /sys/class/gpio/gpio319/value')
        time.sleep(0.3)
        os.system('echo 0 > /sys/class/gpio/gpio319/value')
        time.sleep(0.1)
    elif (wheel_dir_current == left):
        wheel_dir = straight
        os.system('echo 0 > /sys/class/gpio/gpio318/value') # turn right
        time.sleep(0.1)
        os.system('echo 1 > /sys/class/gpio/gpio319/value')
        time.sleep(0.3)
        os.system('echo 0 > /sys/class/gpio/gpio319/value')
        time.sleep(0.1)
    else:
        wheel_dir = wheel_dir_current
        print('Wheels cannot turn any further right...')

    return wheel_dir

def full_turn_right(wheel_dir_current):
    if (wheel_dir_current == left):
        wheel_dir = right
        os.system('echo 0 > /sys/class/gpio/gpio318/value') # turn right
        time.sleep(0.1)
        os.system('echo 1 > /sys/class/gpio/gpio319/value')
        time.sleep(0.6)
        os.system('echo 0 > /sys/class/gpio/gpio319/value')
        time.sleep(0.1)
    else:
        wheel_dir = wheel_dir_current
        print('Wheels cannot make full turn right, try half turn right...')

    return wheel_dir

I just added the above function declarations to the beginning of my modified classify.py script, then modified the code right before the Edge Impulse Runner service is called to set the initial direction of the wheels:

.
.
.
    # set motor direction to forward, can add more decision logic later where appropriate
    os.system('echo 1 > /sys/class/gpio/gpio310/value')

    # disable motor until ML model is running
    os.system('echo 0 > /sys/class/gpio/gpio311/value')

    model = args[0]

    dir_path = os.path.dirname(os.path.realpath(__file__))
    modelfile = os.path.join(dir_path, model)

    print('MODEL: ' + modelfile)

    # default starting wheel position 
    defender_dir = straight

    with ImageImpulseRunner(modelfile) as runner:
        try:
            model_info = runner.init()
.
.
.

And with that, the Defender could now keep moving and simply steer around the jack-o-lantern in its path instead of just stopping.

This was a pretty exciting point to get to because the Defender was a legitimately autonomous robot that could avoid my scary little jack-o-lantern all on its own.

I did discover one final bug however: if I let my modified classify.py script run for too long (10+ minutes), it started behaving weirdly. Weirdly as in sometimes it'd start saying that it saw the jack-o-lantern in frame when it really wasn't or the bounding box location data was all over the place, causing the Defender to frantically steering left/right randomly.

This is either a result of me not providing enough training data in Edge Impulse Studio when I initially trained the ML model, or the fact that I compiled the ML model targeted for the Raspberry Pi 4 since the Kria isn't officially supported yet. I can test this by retraining my ML model with more training data (training data = more photos with/without the jack-o-lantern in them). But I'm going to leave that for another day and just enjoy the win of the Defender initially working for now!

Final Thoughts

While I accumulated quite the list of modifications for the next version of this robot, it reminded me that large projects like these can only be accomplished in incremental revisions like this. The perfect version with every single feature you want just isn't going to be the first thing you build. Start with simple versions of the desire features or just less features to start out with, and iterate/improve from there.

While this version was my journey for 2022, I'm excited to iterate on it for 2023. I want to redesign the Lego build to be a lot more sturdy and elegant with the hobby motor setup. I'm also working to accelerate the Edge Impulse Runner service by offloading it into the Kria's programmable logic via a Vitis accelerated kernel. And as you probably noticed, the power supply for the motors and KR260 board are still their respective wall adapters, so I need to build some sort of trailer for the Defender to haul around its battery power supply.

If you enjoyed this project, watch the 2-part video series to see come to life and the final outcome working in person. (If it's not linked here, I'm still finishing up the edit, but check back within the next week!).