Stardew Valley Villager Recognition and Gift Preferences Bot

In this project, I wanted to focus on an emerging field for improving our overall experience while playing and exploring video games, known as computer vision implementation in video games. We have nearly interminable options while employing computer vision to improve video games, from aim assistance bots for first-person shooter (FPS) games to mining bots for role-playing (RPG) games. Although computer vision implementation in video games is commonly utilized for competitive online games, I decided to employ computer vision in one of my favorite video games - Stardew Valley.

In Stardew Valley, even though I played it more than once already, I still needed to scrutinize guides to remember villagers' gift preferences while trying to increase my friendship with certain characters. Thus, I decided to employ computer vision to recognize villagers and display their birthdays and gift preferences (loves, likes, and hates) automatically while playing Stardew Valley.

Since I wanted this project to be as compact as possible, I chose to use a LattePanda Alpha 864s, a high-performance SBC (single-board computer), has an embedded Arduino Leonardo. To recognize villagers while exploring the Stardew Valley world, I deployed the OpenCV template matching. After detecting villagers successfully, to display their gift preferences and birthdays without interrupting the gaming experience or performance, I used the embedded Arduino Leonardo on the LattePanda Alpha. Arduino Leonardo prints the detected villager information on a 3.5" 320x480 TFT LCD Touch Screen (ILI9488) and notifies the player via a buzzer and a 5mm green LED.

Because I was usually struggling to increase my friendship with these eight characters below, I used the OpenCV template matching to recognize them:

• Abigail
• Emily
• Haley
• Maru
• Penny
• Alex
• Harvey
• Sam

After running this project and obtaining accurate results, it obviates the need for shrewd gift acumen for villagers while playing Stardew Valley. 😃 🎁

Huge thanks to DFRobot for sponsoring this project.

Sponsored products by DFRobot:

⭐ LattePanda Alpha 864s | Inspect

⭐ DFRobot 8.9" 1920x1200 IPS Touch Display | Inspect

Step 1: Recognizing villagers while playing Stardew Valley with OpenCV

I decided to employ template matching to recognize villagers in Stardew Valley while playing the game. Template Matching is a method for searching and finding the location of a template image in a larger (input) image. OpenCV provides a built-in function - cv.matchTemplate() - for deploying template matching. It basically slides the template image over the input image (as in 2D convolution) and compares the patches of the input image against the template image to find the overlapped patches. When the function finds the overlapped patches, it saves the comparison results. If the input image is of size (W x H) and the template image is of size (w x h), the output (results) image will have a size of (W - w + 1, H - h + 1). You can get more information regarding template matching from this guide.

There are several comparison methods implemented in OpenCV, and each comparison method applies a unique formula to generate results. You can inspect TemplateMatchModes to get more information regarding comparison methods, describing the formulae for the available comparison methods.

I developed an application in Python to recognize villagers in Stardew Valley and send their information (gift preferences and birthdays) to the Arduino Leonardo. As shown below, the application consists of one folder and five code files:

• main.py
• windowframe.py
• vision.py
• arduino_serial_comm.py
• pics.py
• /assets
• -- templates (cropped villager poses)
• -- villagers.json

I will elucidate each file in detail in the following steps.

To execute my application to recognize villagers successfully, I installed the required modules on Thonny. If you want, you can use a different Python IDE.

⭐ Download the opencv-python module.

pip install opencv-python

⭐ Download the pywin32 module, which provides access to Windows APIs.

pip install pywin32

Step 1.1: Deriving templates from in-game screenshots and defining villager preferences

First of all, to be able to use template matching to detect villagers in Stardew Valley, I needed to create templates for each villager (character) in my list shown above. Since there are different poses in Stardew Valley for characters, I utilized in-game screenshots to catch unique character poses - sitting, turning right, turning left, going forward, etc. Because I was running Stardew Valley on Steam on LattePanda Alpha, I took screenshots while playing the game on Steam by pressing F12.

Then, I cropped character poses from the in-game screenshots to derive my villager templates.

After collecting templates (cropped character poses) for each villager in my list, I saved them in the assets folder. Then, I created a dictionary (character_pics) containing template paths for each villager and stored it in the pics.py file.

Finally, I created a JSON file (villagers.json) to store gift preferences and birthdays for each villager in my list:

• birthday
• loves
• likes
• hates
• introduction

"Abigail": {
		"birthday" : "Fall 13",
		"loves" : "Amethyst, Banana Pudding, Blackberry Cobbler, Chocolate Cake, Pufferfish, Pumpkin, Spicy Eel",
		"likes" : "Quartz",
		"hates" : "Clay, Holly",
		"introduction" : "Abigail is a villager who lives at Pierre's General Store in Pelican Town. She is one of the twelve characters available to marry. "
	},

...

Step 1.2: Capturing real-time game window data (frames)

I needed to elicit real-time frames (screenshots) from the game window while playing Stardew Valley to employ template matching to recognize the villagers in my list. Even though there are several methods to capture screenshots in Python, I decided to utilize the Windows (Win32) API because it can grasp the screen data for only the game window faster than other methods. Also, it can capture frames (screenshots) of the game window when it is not full-screen.

I created a class named captureWindowFrame in the windowframe.py file to obtain real-time game window data (frames) while playing Stardew Valley and convert the raw frame data to process it with OpenCV.

⭐ Do not forget to install the pywin32 module to access the Windows (Win32) API.

⭐ In the __init__ function, define the handle for the game window, get the window sizes, and adjust them to remove redundant pixels from captured frames.

def __init__(self, window_name):
        # Define the handle for the selected window.
        # Full Screen:
        # self.hwnd = win32gui.GetDesktopWindow()
        self.hwnd = win32gui.FindWindow(None, window_name)
        if not self.hwnd:
            raise Exception('Window Not Found: {}'.format(window_name))
        # Window Sizes:
        window_rect = win32gui.GetWindowRect(self.hwnd)
        self.w = window_rect[2] - window_rect[0]
        self.h = window_rect[3] - window_rect[1]
        # Adjust the window to remove unnecessary pixels:
        self.border_px = 8
        self.titlebar_px = 30
        self.w = self.w - (self.border_px * 2)
        self.h = self.h - self.titlebar_px - self.border_px

⭐ In the get_frames function, capture a new frame from the game window with the Win32 API, convert the raw frame data to process it with OpenCV, then clear the raw frame data.

⭐ Before returning the converted image (frame) data, drop the alpha channel and make the image (img) C_CONTIGUOUS to avoid errors while applying template matching.

def get_frames(self):
        # Get the frame (screenshot) data:
        wDC = win32gui.GetWindowDC(self.hwnd)
        dcObj = win32ui.CreateDCFromHandle(wDC)
        cDC = dcObj.CreateCompatibleDC()
        dataBitMap = win32ui.CreateBitmap()
        dataBitMap.CreateCompatibleBitmap(dcObj, self.w, self.h)
        cDC.SelectObject(dataBitMap)
        cDC.BitBlt((0, 0), (self.w, self.h), dcObj, (self.border_px, self.titlebar_px), win32con.SRCCOPY)
        # Convert the raw frame data for OpenCV:
        signedIntsArray = dataBitMap.GetBitmapBits(True)
        img = np.fromstring(signedIntsArray, dtype='uint8')
        img.shape = (self.h, self.w, 4)
        # Clear the frame data:
        dcObj.DeleteDC()
        cDC.DeleteDC()
        win32gui.ReleaseDC(self.hwnd, wDC)
        win32gui.DeleteObject(dataBitMap.GetHandle())
        # Drop the alpha channel and make the image C_CONTIGUOUS to avoid errors while running OpenCV:
        img = img[...,:3]
        img = np.ascontiguousarray(img)
        # Return:
        return img

⭐ In the get_window_names function, print the list of window names and hex codes if necessary.

def get_window_names(self):
        def winEnumHandler(hwnd, ctx):
            if win32gui.IsWindowVisible(hwnd):
                print(hex(hwnd), win32gui.GetWindowText(hwnd))
        win32gui.EnumWindows(winEnumHandler, None)

Step 1.3: Applying real-time object detection to the captured game window frames

After eliciting real-time game window frames (screenshots) and formatting them appropriately for OpenCV, I applied template matching to the captured frames so as to detect one of the eight villagers (characters) in my list:

• Abigail
• Emily
• Haley
• Maru
• Penny
• Alex
• Harvey
• Sam

I created a class named computerVision in the vision.py file to process the captured real-time game window frames (screenshots) with OpenCV by employing template matching. The class searches for all given poses (templates) of a villager in the captured frames and generates results for each template. Also, it presents the real-time villager detection results by drawing rectangles or markers on the captured frames.

I used the TM_CCOEFF_NORMED comparison method (explained above) to generate results. Since template matching (cv.matchTemplate) generates results in confidence scores for each possible match location in the main (input) image, I define a threshold (0.65) to get the most accurate results for multiple objects (templates). You can examine other comparison methods and different threshold values while applying template matching.

⭐ Do not forget to install the opencv-python module to utilize template matching in OpenCV.

⭐ In the __init__ function, read image data with the cv.imread function from each template (character pose) in the given cropped image list and save template sizes.

def __init__(self, cropped_images_to_detect):
        # Define parameters:
        self.templates = []
        self.template_sizes = []
        # Get and save the required parameters (template and sizes) for all cropped images in the given list:
        for img in cropped_images_to_detect:
            # Template:
            cropped_img = cv.imread(img, cv.IMREAD_UNCHANGED)
            self.templates.append(cropped_img)
            # Sizes:
            cropped_w = cropped_img.shape[1]
            cropped_h = cropped_img.shape[0]
            self.template_sizes.append((cropped_w, cropped_h))

⭐ In the detect_cropped_image function, employ template matching on the given main image (game window frame) to generate results for each given template (character pose):

⭐ Obtain the detected template (character pose) locations above the given threshold (0.65) on the main image (window frame).

⭐ Zip the detected template locations into (X, Y) position tuples.

⭐ Create the rectangles list [x, y, w, h] from the template locations and sizes. Add each element to the rectangles list twice to avoid excluding single points from the list while grouping.

⭐ Group the rectangles list to avert showing overlapping or close positions by the 0.7 relative difference between sides of the given rectangles.

def detect_cropped_image(self, main_img, threshold, method=cv.TM_CCOEFF_NORMED):
        self.main_img = main_img
        # Generate results for each template:
        self.rectangles = []
        for i in range(len(self.templates)):
            # Get detected cropped image locations on the main image:
            result = cv.matchTemplate(self.main_img, self.templates[i], method)
            locations = np.where(result >= threshold)
            # Zip the detected cropped image locations into (X, Y) position tuples:
            locations = list(zip(*locations[::-1]))
            # Create the rectangles list [x, y, w, h] to avert overlapping positions:
            for loc in locations:
                w, h = self.template_sizes[i]
                rect = [int(loc[0]), int(loc[1]), int(w), int(h)]
                self.rectangles.append(rect)
                # For single points:
                self.rectangles.append(rect)
        # Group the rectangles list:
        self.rectangles, weights = cv.groupRectangles(self.rectangles, 1, 0.7)

⭐ In the get_and_draw_points function, loop through all detected template data in the rectangles list to define the rectangle box and center points:

⭐ According to the selected result type, draw rectangles (boxes) around the detected template locations or markers on the center points of the detected template locations.

⭐ Then, return the altered main image (game window frame) with boxes or markers and clear the rectangles list to avoid errors. If nothing is detected, return the original frame.

⭐ Modify the built-in parameters in the function to change the color and size settings of boxes and markers.

def get_and_draw_points(self, result_type, rect_color=(255,0,255), rect_type=cv.LINE_4, rect_thickness=3, marker_color=(255,0,255), marker_type=cv.MARKER_STAR, marker_size=25, marker_thickness=3):
        self.detected_center_points = []
        if(len(self.rectangles)):
            # Loop all detected image locations in the rectangles list:
            for (x, y, w, h) in self.rectangles:
                # Define the rectangle box points:
                top_left = (x, y)
                bottom_right = (x + w, y + h)
                # Define the rectangle center points for drawing markers:
                center_x = x + int(w/2)
                center_y = y + int(h/2)
                # Save the center points:
                self.detected_center_points.append((center_x, center_y))
                # Draw results on the main image:
                if(result_type == 'boxes'):
                    # Draw rectangles around the detected cropped images:
                    cv.rectangle(self.main_img, top_left, bottom_right, color=rect_color, thickness=rect_thickness, lineType=rect_type)
                elif(result_type == 'markers'):
                    # Draw markers on the detected cropped images:
                    cv.drawMarker(self.main_img, (center_x, center_y), marker_color, marker_type, marker_size, marker_thickness)
        return self.main_img
        # Clear the rectangles list to avoid errors:
        self.rectangles = []

⭐ In the get_center_points function, return the center points of the detected template locations.

def get_center_points(self):
        return self.detected_center_points

Step 1.4: Sending the detected villager information to the Arduino Leonardo

As shown above, I created a JSON file (villagers.json) to store gift preferences and birthdays for each villager in my list. After detecting villagers successfully by applying template matching on real-time game window frames, I created a function named arduino_serial_comm in the arduino_serial_comm.py file to send the detected villager information in the villagers.json file to the build-in Arduino Leonardo port via serial communication.

⭐ In the arduino_serial_comm function, define the build-in Arduino Leonardo port (COM6) on the LattePanda Alpha.

⭐ Then, elicit information of the given villager (character) from the villagers.json file and send that information to the Arduino Leonardo via serial communication.

⭐ Do not forget to use the encode function to be able to send strings via serial communication.

def arduino_serial_comm(port, character):
    # Define the data file (json):
    json_file = 'assets/villagers.json'
    # Define the build-in Arduino Leonardo port on the LattePanda:
    arduino = serial.Serial(port, 9600, timeout=1)
    # Elicit information:
    with open (json_file) as data:
        villagers = json.load(data)
        c = villagers[character]
        msg = "Character: {}\n\nBirthday: {}\n\nLoves: {}\n\nLikes: {}\n\nHates: {}".format(character, c['birthday'], c['loves'], c['likes'], c['hates'])
        print(msg)
        arduino.write(msg.encode())
        sleep(1)

Step 1.5: Initiating and running real-time villager recognition

After creating all the code files mentioned above, I decided to combine all required classes and functions in the main.py file to make my code more coherent.

⭐ Include the required classes and functions.

⭐ Define a villager (character) name among the eight villagers in the given list above.

⭐ Create a list of templates (cropped villager poses) of the given character.

⭐ Define the class objects.

⭐ Define the detection status parameter to avoid sending repetitive messages to the Arduino Leonardo.

import cv2 as cv
import numpy as np
from windowframe import captureWindowFrame
from vision import computerVision
from arduino_serial_comm import arduino_serial_comm
from pics import character_pics

# Define a character (villager) name in Stardew Valley:
_character = 'Penny'
# Create a list of cropped images:
cropped_images_to_detect = [character_pics[_character][0], character_pics[_character][1], character_pics[_character][2]]

# Define the class objects:
winframe = captureWindowFrame('Stardew Valley')
game_vision = computerVision(cropped_images_to_detect)

# Define the detection status to avoid repetitive messages to the Arduino Leonardo:
detection_status = True

⭐ In the detect_character function:

⭐ Get a new game window frame (screenshot).

⭐ Detect template locations in the captured game window frame and generate results.

⭐ Obtain the output image and the center points of the detected template locations.

⭐ If template matching recognizes the villager in the given frame, send the detected villager information to the Arduino Leonardo via serial communication. Unless the detected villager disappears in frames, do not send data again to avoid repetitive messages.

⭐ When recognized, save the output image with the name of the detected villager - cv.imwrite.

⭐ Display the output image - cv.imshow.

def detect_character(character, result_type, port):
    global detection_status
    # Get a new frame (screenshot):
    new_frame = winframe.get_frames()
    # Detect the cropped images in the given list:
    game_vision.detect_cropped_image(new_frame, 0.65)
    # Get the output image:
    output = game_vision.get_and_draw_points(result_type)
    # Get the detected cropped image points:
    points = len(game_vision.get_center_points())
    print(points)
    # Send message (data) to the Arduino Leonardo via serial communication:
    if(points == 0):
        detection_status = True
    elif (points > 0 and detection_status):
        detection_status = False
        arduino_serial_comm(port, character)
        # Save the output image:
        cv.imwrite('result_{}.jpg'.format(character), output)
    # Show the output image:
    cv.imshow('OpenCV Computer Vision on LattePanda', output)

⭐ In the loop, initialize the detect_character function.

⭐ Press 'q' to exit and clear.

while True:
    # Initialize villager (character) recognition:
    detect_character(_character, 'boxes', 'COM6')
    #detect_character(_character, 'markers', 'COM6')
    
    # Press 'q' to exit:
    if(cv.waitKey(1) == ord('q')):
        cv.destroyAllWindows()
        break

After running the main.py file successfully:

🎮💻 It opens a new window named OpenCV Computer Vision on LattePanda to show real-time captured game window frames (screenshots) and detection results as a video stream.

🎮💻 Then, according to the selected result type, it draws rectangles (boxes) around the detected template locations or markers on the center points of the detected template locations.

🎮💻 Finally, it displays the number of the detected template locations in the given game window frame on the shell and sends the recognized villager information to the Arduino Leonardo via serial communication. As explained above, it sends data for once until the detected villager disappears in frames to avoid repetitive messages.

Step 2: Displaying the transferred villager information with the Arduino Leonardo

After completing all steps above and transferring the detected villager information successfully to the embedded Arduino Leonardo on the LattePanda Alpha via serial communication, I utilized a 3.5" 320x480 TFT LCD Touch Screen (ILI9488) to display that information. Also, I used a buzzer and a 5mm green LED to notify the player when template matching recognizes a villager (character).

Download the required libraries to print data on the 320x480 TFT LCD Touch Screen (ILI9488):

ILI9488 | Library

Adafruit_GFX | Library

⭐ Include the required libraries.

#include "SPI.h"
#include <Adafruit_GFX.h>
#include <ILI9488.h>

⭐ Define the required pins for the 320x480 TFT LCD Touch Screen (ILI9488) and initiate it with the hardware SPI on the Arduino Leonardo (SCK, MISO, MOSI).

#define TFT_DC 9
#define TFT_CS 10
#define TFT_RST 8

// Use hardware SPI (on Leonardo, SCK, MISO, MOSI) and the above for DC/CS/RST
ILI9488 tft = ILI9488(TFT_CS, TFT_DC, TFT_RST);

⭐ Initialize serial communication and the 320x480 TFT LCD Touch Screen (ILI9488).

Serial.begin(9600);
  // Initialize the TFT LCD Touch Screen (ILI9488):
  tft.begin();
  tft.setRotation(1);
  tft.fillScreen(ILI9488_RED);
  tft.setCursor(0, 0);
  tft.setTextColor(ILI9488_BLACK); tft.setTextSize(4);
  tft.println("\nStardew Valley\n");
  tft.println("Villager Recognition\n");
  tft.println("and Gift Preferences\n");
  tft.println("Bot w/ OpenCV\n");

⭐ If there is incoming serial data, save it to the gameData string to obtain the information of the villager detected by template matching.

while (Serial.available()) {
    char c = (char)Serial.read();
    gameData += c;
  }

⭐ After eliciting the detected villager information, display that information (character name, birthday, loves, likes, hates) on the TFT LCD Touch Screen and notify the player via the buzzer and the 5mm green LED. Then, clear the gameData string.

if(gameData != ""){
    // Notify the user:
    tft.fillScreen(ILI9488_BLACK);
    tft.setCursor(10, 5);
    tft.setTextColor(ILI9488_RED); tft.setTextSize(4);
    tft.println("Stardew Valley\n");
    tft.setTextColor(ILI9488_GREEN); tft.setTextSize(2);
    tft.println(gameData);
    tone(buzzer, 500);
    digitalWrite(green_LED, HIGH);
    delay(500);
    noTone(buzzer);
    digitalWrite(green_LED, LOW);
  }
  // Clear:
  delay(250);
  gameData = "";

After running this code (lattepanda_detection_results_display.ino) on the embedded Arduino Leonardo on the LattePanda Alpha:

🎮💻 It shows the home screen while waiting for incoming serial data.

🎮💻 After template matching detects a villager (character) and transfers the detected villager information to the Arduino Leonardo via serial communication, it displays:

• Character (Name)
• Birthday
• Loves
• Likes
• Hates

🎮💻 Also, it notifies the player via the buzzer and the 5mm green LED.

Results for each villager while exploring Stardew Valley

After completing my project, I explored the Stardew Valley world to detect villagers (characters) in my list with template matching and display their information (birthdays and gift preferences) on the Arduino Leonardo:

• Abigail
• Emily
• Haley
• Maru
• Penny
• Alex
• Harvey
• Sam

As far as my experiments go, template matching runs stupendously on real-time captured game window frames from Stardew Valley:

📌 Abigail

📌 Emily

📌 Haley

📌 Maru

📌 Penny

📌 Alex

📌 Harvey

📌 Sam

Connections and Adjustments

// Connections
// LattePanda Alpha 864s (Arduino Leonardo) :  
//                                3.5'' 320x480 TFT LCD Touch Screen (ILI9488)
// D10 --------------------------- CS 
// D8  --------------------------- RESET 
// D9  --------------------------- D/C
// MOSI -------------------------- SDI 
// SCK --------------------------- SCK 
// MISO -------------------------- SDO(MISO) 
//                                Buzzer
// D11 --------------------------- S     
//                                5mm Green LED
// D12 --------------------------- S

I connected the 3.5" 320x480 TFT LCD Touch Screen (ILI9488), the buzzer, and the 5mm green LED to the embedded Arduino Leonardo via the LattePanda Alpha GPIO pins.

Since the Arduino Leonardo operates at 5V and the TFT LCD Touch Screen (ILI9488) requires 3.3V logic level voltage, I utilized two bi-directional logic level converters to shift the voltage for the connections between the TFT LCD Touch Screen (ILI9488) and the Arduino Leonardo.

Videos and Conclusion

After completing all steps above, I managed to increase my friendship with Stardew Valley villagers (characters) effortlessly without needing to scrutinize guides so as to remember a certain villager's gift preferences and birthday 😃