Published April 10, 2020 © MIT

Deep Learning Sumo Robot

Hajimete! Let DL pushing madness begin!

IntermediateProtip3 hours2,205

Things used in this project

Hardware components

Seeed Studio M.A.R.K kit

Software apps and online services

aXeleRate

Story

Robot-sumo, or pepe-sumo, is a sport in which two robots attempt to push each other out of a circle (in a similar fashion to the sport of sumo). The robots used in this competition are called sumobots.

From Wikipedia, the free encyclopedia

UPDATED 03/29/2022. I try my best to keep my articles updated on a regular basis and based on your feedback from YouTube/Hackster comments section. If you'd like to show your support and appreciation for these efforts, consider buying me a coffee (or a pizza) :) .

Robot sumo is one of the classic and most popular robot competitions. There is a huge variety of builds used to accomplish the task of pushing the opponent out of the ring.

Example of a sumo robot pushing an opponent out of the arena Retrieved from http://www.ourgemcodes.com/japanese-pepe-sumo-robotics-sport-incredible-speeds/

Many simpler robots rely on ultrasonic or infrared sensors to find the opponent before taking offensive action. In this article we will use M.A.R.K. mobile platform to make a variation of sumo bot.

M.A.R.K. (I'll call it MARK in text) stands for Make a Robot Kit and it is an educational robot platform in development by TinkerGen education. I take part in the development of MARK and we’re currently preparing to launch a Kickstarter campaign. In accompanying courses the students will learn how to complete a variety of tasks with MARK, i.e. self-driving, patrol, delivery service, etc. When I was writing course materials for High School M.A.R.K. course I wanted to include a challenge in the course that was both familiar to STEM teachers and students and at the same time had a unique twist. So I decided to design a robot sumo competition, where two MARK robots had to use DL models for detecting an opponent. This way two factors decide whose robot is more likely to win:

the accuracy of trained model
the algorithm of offensive action after detection is confirmed

We will use aXeleRate, Keras-based framework for AI on the Edge and the model training pipeline is very similar to what we had before with person detector. The only problem we have in this case is lack of suitable dataset - doesn't matter if you want to do sumo competition with your own custom robot or MARK, there are no readily available datasets to download on the internet. Creating object detection dataset from scratch is a tedious task, usually at least 1000 pictures needed for one class to achieve acceptable results.

Fortunately with MARK we can use a shortcut. MARK chassis is mostly black :) So we can write a simple OpenCV script that would detect biggest black blob in the image and draw a bounding box around it.

The results are not perfect, but easier than annotating from scratch

Then we would process all the images we taken with smartphone camera of the robot in at least 4-5 different environments. The annotation files we get from OpenCV script have some errors (especially if the environment is dark with a lot of shadows) and we will need to verify and correct them in labelImg annotation tool before using in training.

https://github.com/tzutalin/labelImg

After the dataset is ready, let's use this Colab notebook to train the model. I also share baseline trained model, which is included with the course for students to get the idea of what the normal performance of the model should be. The full Micropython code for MARK is in code section of this article. Have a look at final result videos - one in the head of the article and one below. Since it is a student competition, both model and robot code can (should :) ) be improved to get the edge over opponents.

Add me on LinkedIn if you have any questions and subscribe to my YouTube channel to get notified about more interesting projects involving machine learning and robotics.

Stay tuned for more articles from me and updates on MARK Kickstarter campaign!

import sensor,image,lcd, os, time
from maix_motor import Maix_motor
import KPU as kpu

lcd.init()
sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QVGA)
sensor.set_windowing((224, 224))
sensor.set_vflip(1)
sensor.run(1)
DEBUG = 0
classes = ["mark"]
task = kpu.load(0x200000)
a = kpu.set_outputs(task, 0, 7,7,30)
anchor = (0.57273, 0.677385, 1.87446, 2.06253, 3.33843, 5.47434, 7.88282, 3.52778, 9.77052, 9.16828)
a = kpu.init_yolo2(task, 0.6, 0.3, 5, anchor)
while(True):
    img = sensor.snapshot().rotation_corr(z_rotation=90.0)
    a = img.pix_to_ai()
    code = kpu.run_yolo2(task, img)
    if code:
        for i in code:
            a=img.draw_rectangle(i.rect(),color = (0, 255, 0))
            a = img.draw_string(i.x(),i.y(), classes[i.classid()], color=(255,0,0), scale=3)
            x_center = i.x()+i.w()/2
            print(x_center)
            if not DEBUG:
                if x_center >= 100 and x_center <= 124:
                    while 1:
                        Maix_motor.motor_motion(3, 1, 0)
                        time.sleep(1.5)
                        Maix_motor.motor_motion(3, 3, 0)
                        time.sleep(0.1)
                        Maix_motor.motor_motion(3, 4, 0)
                        time.sleep(0.1)
                if x_center < 100 and x_center > 0: Maix_motor.motor_motion(1, 4, 0)
                if x_center > 124: Maix_motor.motor_motion(1, 3, 0)
        a = lcd.display(img)
    else:
        a = lcd.display(img)
        if not DEBUG: Maix_motor.motor_motion(1, 3, 0)
a = kpu.deinit(task)

# Standard imports
import cv2
import numpy as np;
import os
from pascal_voc_writer import Writer

def create_ann(filename, boundRect):
    writer = Writer(os.path.join('Mark',filename), 3000, 4000)
    writer.addObject('mark', boundRect[0], boundRect[1], boundRect[0]+boundRect[2], boundRect[1]+boundRect[3])
    name = filename.split('.')
    writer.save('ann/'+name[0]+'.xml')

for file in os.listdir("Mark"):
    print(file)
    img = cv2.imread(os.path.join("Mark",file))
    R,G,B = cv2.split(img)

    #Rfilter = cv2.bilateralFilter(G,25,25,10)

    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

    ret, thresh = cv2.threshold(gray,50,255,cv2.THRESH_BINARY_INV)

    contours, hierarchy = cv2.findContours(thresh,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_NONE)

    finalImage = cv2.drawContours(img, contours, -1,(0,0,255),3)

    maxContour = 0
    for contour in contours:
        contourSize = cv2.contourArea(contour)
        if contourSize > maxContour:
            maxContour = contourSize
            maxContourData = contour

    contours_poly = cv2.approxPolyDP(maxContourData, 3, True)
    boundRect = cv2.boundingRect(contours_poly)
    print(boundRect)

    # Create a mask from the largest contour
    mask = np.zeros_like(thresh)
    cv2.fillPoly(mask,[maxContourData],1)

    # Use mask to crop data from original image
    finalImage = np.zeros_like(img)
    finalImage[:,:,0] = np.multiply(R,mask)
    finalImage[:,:,1] = np.multiply(G,mask)
    finalImage[:,:,2] = np.multiply(B,mask)

    print(finalImage.shape)
    cv2.rectangle(finalImage, (int(boundRect[0]), int(boundRect[1])), (int(boundRect[0]+boundRect[2]), int(boundRect[1]+boundRect[3])), (0,255,0),3)

    finalImage = cv2.resize(finalImage,(640,480))

    cv2.imshow('final',finalImage)
    cv2.waitKey(50)
    create_ann(file, boundRect)
cv2.destroyAllWindows()

Credits

Dmitry Maslov

31 projects • 180 followers

Deep Learning Sumo Robot

Things used in this project

Hardware components

Software apps and online services

Story

Code

MARK micropython sumo code

OpenCV blob detection for annotations

Credits

Dmitry Maslov

Comments

Embed the widget on your own site

Deep Learning Sumo Robot

Deep Learning Sumo Robot

Things used in this project

Hardware components

Software apps and online services

Story

Code

MARK micropython sumo code

OpenCV blob detection for annotations

Credits

Dmitry Maslov

Comments

Related channels and tags