The gesture recognition game(rock, paper, scissors) example uses machine learning technique to identify the player's gestures and play against HUB-8735. During the game, the TFT LCD will display the player's gestures and display the result of victory or defeat.
About HUB 8735HUB 8735, a full Taiwan-Made IoT Board, using Realtek's Ameba RTL8735 module as main MCU.
HUB 8735 Smart AI CAM is a highly integrated module with multi-functional image processing. It has a built-in NPU AI computing engine to accelerate the processing of AI models and 802.11 a/b/g/n dual-band Wi-Fi and BLE low-power Bluetooth transmission. It is widely used in various IoT fields that combine image recognition or AI computing, and is suitable for scenarios such as smart homes, industrial IoT, smart retail, health care, or automotive electronics.
Many pre-trained AI models(such as, Yolov3-tiny, Yolov4-tiny, Yolov7-tiny, etc.) have been optimized, and capable to run directly on the module and can be used as a tool for STEAM education. At the same time, with a size of only 30.5x43.9mm, it can also be directly integrated into product design as a quick-production kit for quickly importing Edge AI applications.
The main features of HUB 8735 include:
- Using Ameba RTL8735
- Compatible with Arduino development features
- Highly integrated module with multi-functional image processing
- Built-in NPU AI computing engine accelerates processing of AI models
- Provides 802.11 a/b/g/n dual-band Wi-Fi and BLE low-power Bluetooth transmission
- Can be widely used in various IoT fields combined with image recognition or AI computing
HUB 8735 can be developed on Arduino. In order to do so, we need to add additional board information to Arduino IDE.
Add the below URL to your Arduino's additional board manger in File>Preferences:
https://github.com/ideashatch/HUB-8735/raw/main/amebapro2_arduino/Arduino_package/ideasHatch.json
Next, we can search for the board and install it in Board Manger.
After that, we can select the board in Tools, and a lots of built-in examples you can try out!
In this project we had yolov4-tiny model(Darknet-based) trained on Google Colab with 370 images of rock, paper, scissors. We have the model's weights and config file on our Github , even the dataset, code too, please feel free to explore it! We are trying not to discuss the process of training a model here, since it's a bit dull(just kidding!). We strongly encourage you to explore the code in the Github repo, it's easy get started, and clarify for understanding.
Well, the weights can't directly port into any board, it has to be converted to binary file. Thanks to Realtek, we can easily convert it online(HERE). You'll have to create an account to use the converting tool. There are several models with different frameworks supported, you'll need to adjust the required files for the conversion.
You'll have to zip.cfg file and weights file before uploading it, and required at least one test image file!
Await 1 to 5 minutes, you'll be receiving the.nb file in your provided email, and download it!
Now we have the magic.nb file, where should we put it? Since we originally develop in Windows, we will demonstrate for Windows(Linux will be the same, just different path, I'm sure you're capable to find it). First, we gonna go to the folder path shown below:
C:\Users\<USER>\AppData\Local\Arduino15\packages\ideasHatch\hardware\AmebaPro2\<BUILD_VERSION>\variants\common_nn_models
If you can't find it, remember to check the file explorer's view settings, make sure hidden items is checked. Once you found it, our folder should look like this with several models' binary files.
The converted.nb file from Realtek, you can directly put it here, and rename it to yolov4_tiny.nb
You can store the built-in binary files elsewhere just in case you don't want to replace it.
Time for the game!If you visited the Github repo, you'll surely find the code in src
folder. Open the code in Arduino and you will see the whole code and some headers file, and you can check for the ObjectClassList.h
where classes' names are stored.
Before trying it out(if you happen to have one of our board), remember to change the Wi-Fi SSID & Password in line 72&73.
In this project, we use a TFT LCD ILI9341 for display purpose, you can refer to the schematic diagram below for the wire connections. This LCD screen is displaying the result of the game, along with your move and your opponent(HUB 8735), you can try your best to beat the machine now.
To assure that you are in the range of effective capture range, the code/program is able to establish RTSP streaming connection, which is already written in the code.
We recommend using the free open-source media player -> VLC media
player
as it is easy to use. But first, we have to find and copy the RTSP address in serial monitor. Once, you found it, open VLC and paste the address(Media > Open Network Stream). Note that, your computer and HUB 8735 must be in the same network!
It will take a while to establish the connection, once it is done, it will show something like this:
In the other hand, you can start playing with the board!
Brief code explanationThe ObjDet.getResultCount()
from the helpers function get the neural network's result, and this is mainly where we can retrieve the object index in ObjectClassList.h
. Then, the TFT LCD will display the move for the player's gesture.
The board(or machine) will generate random number for the move, but don't worry it's not cheating! Since this is just a pretty easy demonstration, we didn't design the draw situation, only win or lose situation(okay, we are just lazy, please don't mind). Feel free to improvise something else!
// Generate random move
myNum = random(3);
// Winning message
tft.setForeground(ILI9341_GREEN);
tft.print("YOU WIN");
ConclusionIn this project, we demonstrated a Rock, Paper, Scissors game using HUB 8735, a Smart AI Camera IoT board with built-in NPU AI engine. There is still room for the improvement, such as different situation. But most importantly, this is extremely convenient IoT board, that can use its camera and NPU to do a lots of applications. Hopefully, this project will inspire you!
More detail: HUB 8735 Github
(This is the first time we are sharing our project here, so if you have anything just hit us up, we will improve it! Thanks!)
Comments
Please log in or sign up to comment.