M5Stack synchronizing the colors with Unity

I recently started Unity. I usually play with electronic works, so I am still copying a beginner's book, but I investigated how to communicate Unity and other devices wirelessly.

Mainly when I checked it, it seemed that UDP and WebSocket could be used, so this time I tried to communicate with Unity-M5Stack using WebSocket. M5Stack is a gadget that is equipped with a module called ESP32 that can communicate with WiFi and is convenient for prototyping with an LCD and buttons.

The result is as per this Tweet.

I assigned the buttons of M5Stack to red, green and blue, and when I pressed the button, the color of Cube on Unity and the screen of M5Stack changed.

In the end, I was able to connect multiple M5 series (M5Stack, M5StickC, M5Atom) to a PC running Unity and synchronize multiple colors at the same time.

System configuration

As a configuration, there is Unity Project on the PC side, and WebSocket Server and Client are placed there. The function of WebSocket Client is put in M5 Series which communicates with WiFi from the outside.

Unity WebSocket-Sharp

When using WebSocket with Unity, use the library called WebSocekt-Sharp.

Please refer to the usage guide as it is summarized here.

Using WebSocket with Unity

First, I prepared WebSocket Server and Client within Unity as in the above article, and confirmed whether communication could be performed between them.

ArduinoWebSocket

The M5Stack side uses the Arduino IDE, so I used this library.

arduinoWebSockets

M5Stack side is used as WebSocket Client. Click here for an example.

https://github.com/Links2004/arduinoWebSockets/blob/master/examples/esp32/WebSocketClient/WebSocketClient.ino

I will write a code to synchronize colors based on these.

M5Stack (WebSocket Client)

Source code.

#include <WiFi.h> 
#include <WebSocketsClient.h> 
#include <ArduinoJson.h> 
#include <M5Stack.h> 
#include <map> 
#include "config.h"

WebSocketsClient webSocket;
DynamicJsonDocument doc(1024);

std::map<std::string, uint32_t> colorMap{
   {"red", RED},
   {"green", GREEN},
   {"blue", BLUE}
};

std::string parseReceivedJson(uint8_t *payload)
{
 char *json = (char *)payload;
 DeserializationError error = deserializeJson(doc, json);
 
 if (error) {
   Serial.print(F("deserializeJson() failed: "));
   Serial.println(error.c_str());
   return "none";
 }

 JsonObject obj = doc.as<JsonObject>();

 // You can use a String to get an element of a JsonObject
 // No duplication is done.
 return obj[String("color")];
}

void syncColor(uint8_t *payload)
{
 std::string color = parseReceivedJson(payload);

 Serial.printf("color: %s\n", color.c_str());
 //M5.Lcd.fillRect(60, 20, 200, 200, colorMap[color]);
 M5.Lcd.fillScreen(colorMap[color]);
}

void webSocketEvent(WStype_t type, uint8_t * payload, size_t length) {

  switch(type) {
    case WStype_DISCONNECTED:
      Serial.printf("[WSc] Disconnected!\n");
      break;
    case WStype_CONNECTED:
      Serial.printf("[WSc] Connected to url: %s\n", payload);
      //webSocket.sendTXT("Connected");
      break;
    case WStype_TEXT:
      Serial.printf("[WSc] get text: %s\n", payload);
            syncColor(payload);
      break;
    case WStype_BIN:
    case WStype_ERROR:      
    case WStype_FRAGMENT_TEXT_START:
    case WStype_FRAGMENT_BIN_START:
    case WStype_FRAGMENT:
    case WStype_FRAGMENT_FIN:
      break;
  }
}

void setupWiFi()
{
 WiFi.begin(ssid, passwd);

 // Wait some time to connect to wifi
 for(int i = 0; i < 10 && WiFi.status() != WL_CONNECTED; i++) {
     Serial.print(".");
     delay(1000);
 }

 // Check if connected to wifi
 if(WiFi.status() != WL_CONNECTED) {
     Serial.println("No Wifi!");
     return;
 }

 Serial.println("Connected to Wifi, Connecting to server.");
  // server address, port and URL
  webSocket.begin("192.168.10.11", 8080, "/");

  // event handler
  webSocket.onEvent(webSocketEvent);

  // use HTTP Basic Authorization this is optional remove if not needed
  //webSocket.setAuthorization("user", "Password");

  // try ever 5000 again if connection has failed
  webSocket.setReconnectInterval(5000);
}

void setup()
{
 Serial.begin(115200);
 // Power ON Stabilizing...
 delay(500);
 M5.begin();

 setupWiFi();
 M5.Lcd.fillScreen(BLACK);
 M5.Lcd.setTextColor(GREEN);
 M5.Lcd.setTextSize(2);
}

void loop() {
 bool isPressedBtnA = false;
 bool isPressedBtnB = false;
 bool isPressedBtnC = false;
 if(M5.BtnA.wasPressed() || M5.BtnA.isPressed())
 {
   isPressedBtnA = true;
 }
 if(M5.BtnB.wasPressed() || M5.BtnB.isPressed())
 {
   isPressedBtnB = true;
 }
 if(M5.BtnC.wasPressed() || M5.BtnC.isPressed())
 {
   isPressedBtnC = true;
 }

 static uint32_t pre_send_time = 0;
 uint32_t time = millis();
 if(time - pre_send_time > 100){
   pre_send_time = time;
   String isPressedBtnAStr = (isPressedBtnA ? "true": "false");
   String isPressedBtnBStr = (isPressedBtnB ? "true": "false");
   String isPressedBtnCStr = (isPressedBtnC ? "true": "false");
   String btn_str = "{\"red\":" + isPressedBtnAStr + 
     ", \"green\":" + isPressedBtnBStr + 
     ", \"blue\":" + isPressedBtnCStr + "}";
   //Serial.println(btn_str);
   webSocket.sendTXT(btn_str);
 }
 webSocket.loop();

 M5.update();
}

setupWiFi() adds to the common ESP32 WiFi connection process.

WebSocket.begin() handles WebSocekt connection. The specified IP and PORT will be the connection destinations on the PC side, so change them appropriately. In addition, Port may not be opened depending on the firewall settings and security software settings on the PC side. In that case, each setting is required. I will omit it here.

The button press is determined in loop(), and if the button is pressed, the push determination is set to true and JSON is generated and sent with webSocket.sendTXT(btn_str).

For example, if only the red button (left button) is pressed, this JSON will be sent.

{"red":true, "green":false, "blue":false}

This will send a color button event to the server.

Unity (WebSocket Server)

Source code for Server side.

using UnityEngine;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using WebSocketSharp;
using WebSocketSharp.Net;
using WebSocketSharp.Server;

public class ColorStatusServer: MonoBehaviour {

   WebSocketServer server;
   Queue<string> messages = new Queue<string>();
   Queue<string> colorSetting = new Queue<string>();
   public ColorButtonStatus colorButtonStatus = new ColorButtonStatus();


   [System.Obsolete]
   void Start ()
   {

       server = new WebSocketServer(8080);

       server.AddWebSocketService<ColorStatus>("/", () => new ColorStatus(){
           Messages = messages,
           ColorSetting = colorSetting
       });
       server.Start();

   }

   void Update() {
       if(messages.Count() > 0) {
           string recv_msg = messages.Dequeue();
           //Debug.Log("recv:" + recv_msg);  
           colorButtonStatus = JsonUtility.FromJson<ColorButtonStatus>(recv_msg);

           //Debug.Log("red:" + colorButtonStatus.red);
           //Debug.Log("green:" + colorButtonStatus.green);
           //Debug.Log("blue:" + colorButtonStatus.blue);

          // MeshRendererのメンバであるマテリアルの色を変更
          if (colorButtonStatus.red == true) {
               transform.GetChild(0).gameObject.GetComponent<Renderer>().material.color = Color.red;
               colorSetting.Enqueue("red");
          } else if (colorButtonStatus.green == true) {
               transform.GetChild(0).gameObject.GetComponent<Renderer>().material.color = Color.green;
               colorSetting.Enqueue("green");
          } else if (colorButtonStatus.blue == true) {
               transform.GetChild(0).gameObject.GetComponent<Renderer>().material.color = Color.blue;
               colorSetting.Enqueue("blue");
          }
       }
   }

   public void ClearColor()
   {
       transform.GetChild(0).gameObject.GetComponent<Renderer>().material.color = Color.white;
   }

   void OnDestroy()
   {
       server.Stop();
       server = null;
   }

}

public class ColorStatus : WebSocketBehavior
{
   public Queue<string> Messages;
   public Queue<string> ColorSetting;

   public void SendMessage(string msg)
   {
       Sessions.Broadcast(msg);
   }

   protected override void OnMessage (MessageEventArgs e)
   {
       Messages.Enqueue(e.Data);

       ColorButtonStatus buttonStatus = JsonUtility.FromJson<ColorButtonStatus>(e.Data);

       string color = "{\"color\":\"";
       if(ColorSetting.Count > 0)
       {
           color += ColorSetting.Dequeue();
           color += "\"}";
           Sessions.Broadcast(color);
       }

   }

   
}

The ColorStatusServer class processes GameObject on the Unity side, and the ColorStatus class only performs WebSocekt communication. Data is exchanged between the classes using Queue<string> messages and Queue<string>colorSetting.

OnMessage() is called when JSON of color button comes from WebSocket Client in ColorStatus class.

Messages.Enqueue(e.Data);

Then, pass the data to the ColorStatusServer side, and the fetching is done on the Update side. (Because this is asynchronous, Queue will overflow if WebSocekt is received too early.)

Change the color of Cube using Renderer with Update() of ColorStatusServer. Also, update the colorSetting for broadcasting to the M5 side.

JSON is like this because Server only needs to send the color to change to Cient. (For red)

{"color":red}

Also, looking at the source on the Client side, the color setting received from the Server is set with syncColor() in a method called webSocketEvent().

void syncColor(uint8_t *payload)
{
 std::string color = parseReceivedJson(payload);

 Serial.printf("color: %s\n", color.c_str());
 //M5.Lcd.fillRect(60, 20, 200, 200, colorMap[color]);
 M5.Lcd.fillScreen(colorMap[color]);
}

Since M5Stack can set the color of the entire screen with M5.Lcd.fillScreen(), this switches the color.

Connection of multiple units

As you can see from the code, the Client side is a code that only considers the Server, so you can connect to the Server if you start all M5Stack with exactly the same code. (M5StickC and Atom handle buttons differently, so only those changes need to be made.)

Since the Server side is not aware of each Client, the same process can be performed even if multiple Clients are connected without changing the code. This time it's okay to just broadcast the color change to the Client, but if you want to change the control individually for each Client, you need to determine which Client is connected and divide the processing. It seems that the connection order can be obtained with the function of WebSocekt-Sharp. (Not confirmed if it actually works)

Summary

For more details, check out the code on GitHub.

Click here for M5 side