Published April 6, 2020

Python Dino

Playing Google Chrome Dino game automatically with image processing and simulating key strokes with pynput module

BeginnerProtip15 minutes1,239

Story

Recently I started learning image processing using openCV. so I wanted to have fun with it that's why I created this project.it's easy and works good not perfect but good.

the idea is to detect the cactus in the image by openCV and jump by simulating key strokes using the pynput module.

Now let's try it out :

i'm using windows 10 but these steps are the same for any os that can run python

i'm assuming that you have python already installed on your machine

run these commands in your cmd/terminal :

first install openCV

pip install opencv-python

install pil

pip install Pillow

install pynput

pip install pynput

now we are ready to go

first a new tab in google chrome and make sure that the window is half the size of your screen (ore close to that value). try to put the script on your desktop it will make things easier

now type the following in your search bar :

chrome://dino

this will open the game.

please note that in order to work the game has to be in a specific part of your screen as this code captures a small part and process it.

now run the script and start the game py pressing SPACE key

I added some comments in the code to explain things but if you have any questions feel free to contact me.

a very good source to learn about opencv : https://docs.opencv.org/3.4.6/d6/d00/tutorial_py_root.html

dino.py

import numpy as np
import cv2

#for screen capture
from PIL import ImageGrab

#keyboard
from pynput.keyboard import Key, Controller
keyboard = Controller()


#threshold values
low_black = np.array([0,0,0])
high_black = np.array([180,255,100])

while 1:
    #this is a larger image to see the highlighted contours of obstacles
    #it's just for show

    full_img=ImageGrab.grab(bbox=(100, 100, 640, 480)) #x, y, w, h
    full_frame = np.array(full_img)
    # note : the image is at RGB format but openCV work with BGR so.....
    full_frame = cv2.cvtColor(full_frame, cv2.COLOR_RGB2BGR)


    #the small part of the image that we will check the obstacles in it
    img = ImageGrab.grab(bbox=(100, 100, 250, 345)) #x, y, w, h
    frame = np.array(img)
    #let's go to hsv plane for thresholding :
    hsv_frame = cv2.cvtColor(frame, cv2.COLOR_RGB2HSV)
    #thresholding the image with the ranges determined at the begining of the code
    black_mask = cv2.inRange(hsv_frame, low_black, high_black)

    cv2.imshow("lense", black_mask)

    #ok sooo contours are like the frame that bounds a specific region but if you've noticed,
    #the ground has the same color as the cactus so the program will detect it as an obstacle

    #erodion is simply shrinking the image a liitle bit so that's how we seperate the cactus from the ground
    #dilation is the opposite of erossion, it's used to make the image smoother
    kernel = np.ones((3, 3), np.uint8)
    black_mask = cv2.erode(black_mask, kernel, iterations=1)
    black_mask = cv2.dilate(black_mask, kernel, iterations=9)


    #let's get the contours :
    im2, contours, hierarchy = cv2.findContours(black_mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    if len(contours)>0 :
        cv2.drawContours(frame, contours, -1, (0, 255, 0), 3)

        cactus = max(contours, key = cv2.contourArea)
        x, y, w, h = cv2.boundingRect(cactus)

        if y>160:
            cv2.drawContours(frame, [cactus], -1, (0, 255, 0), 3)
            cv2.rectangle(full_frame, (x, y), (x + w+50, y + h+50), (0, 250, ), 3)

            #the left of the cactus reached a certain point (try to change that value perhaps you would get better results
            if x<42 :
                keyboard.press(Key.up)
                keyboard.release(Key.up)

    cv2.imshow("frame", full_frame)
    key = cv2.waitKey(1)
    if key == 27:
        cv2.destroyAllWindows()
        break