GloMusic

Introduction

Innovation on music instrumentts as an educational and creative tool has being a subject of research in the last couple of years. Having two members of our team being students from a high school with a strong musical curriculum we come up with this new instrument.

We present a new take on music interaction where a single interface (glove) can control different instruments at a time.

The Matrix Voice ESP32 give us the chance to break the process in two different tasks:

• The collection of data using external sensors (pressure) and
• the voice command recognition with SNIPS, instrument management and play.

The ESP32 running a Tensilica Xtensa dual-core microcontroller handle the analog input coming from the pressure sensors and communicate to the Raspberry Pi 3 controller program thru a UART channel.

Materials

The following are snapshots of the materials we used. Note that the resistors are 3.3 Kohms, and that the glove shown does not have the double side tape adhere yet.

We also use two SD cards for the RPi. The first one had the ESP32 programming environment, and the second has the Matrix Core, HAL and Lite packages installed with the python programming support.

The Pressure Sensors

The core data collection to create music is the pressure sensor attached to the device. We needed fast responsiveness on multiple sensors attached simultaneously.

We use the ESP32 expose IO ports from the extension GPIO port on the Matrix Voice ESP32 as analog inputs from the pressure sensors. The Arduino script defines the pins to be activate as follows

//Use extension port for pressure sensors
const int FSR_PIN = 12;  // Pin connected to FSR/resistor divider
const int FSR_PIN1 = 26; // Pin connected to FSR/resistor divider
const int FSR_PIN2 = 27; // Pin connected to FSR/resistor divider
const int FSR_PIN3 = 25; // Pin connected to FSR/resistor divider

The above pins can be physically found in the expansion port as shown below. We also use the 3.3V and GND pin to provide power to the sensors.

Initialize the UART serial port for data transfer and set the ESP32 pins for input in the setup function of the Arduino script.

void setup() {
    Serial.begin(115200);
    pinMode(FSR_PIN, INPUT);
    pinMode(FSR_PIN1, INPUT);
    pinMode(FSR_PIN2, INPUT);
    pinMode(FSR_PIN3, INPUT);

The main loop reads the sensors input. In some situations we have notice that the pins returns a shadow value when read on consecutive instructions. We try adding a timer to mitigate this issue without success and the one that worked better was checking the value of the next pin and if it was the same reading it again. This at some degree fix the issue for us.

void loop()
{
    byte inCmd;
    int fsrADC = analogRead(FSR_PIN);
    int fsrADC1 = analogRead(FSR_PIN1);
    if (fsrADC == fsrADC1)
        fsrADC1 = analogRead(FSR_PIN1);
    int fsrADC2 = analogRead(FSR_PIN2);
    if (fsrADC1 == fsrADC2)
        fsrADC2 = analogRead(FSR_PIN2);
    int fsrADC3 = analogRead(FSR_PIN3);

Now it is time to program the ESP32 chip. It did took us some time to figure out the process and after following the instructions on Program Over the Air on ESP32 MATRIX Voice w/ Arduino IDE we understood the process to flash the program manually. We have to say that Over The Air (OTA) did not work for us, our matrix voice keep crashing at startup. But, manually was working so every time we change the program we just call the deploy_ota, sh script.

Also we use a clean SD card with a fresh Raspbian Stretch Desktop image installed. We followed the instructions on installing ESP32 up to step 1.

In summary, you need to

• Select the ESP32 dev Module board in the Arduino IDE

• Click on the Export Compiled Binary under the Sketch menu. This will create a bin file

• Update the deploy_OTA.sh file with your credentials and the name of the bin file.

tar cf - *.bin | ssh pi@YOURIPADDRESSHERE 'tar xf - -C /tmp;sudo voice_esp32_reset;voice_esptool --chip esp32 --port /dev/ttyS0 --baud 115200 --before default_reset --after hard_reset write_flash -u --flash_mode dio --flash_freq 40m --flash_size detect 0x1000 /tmp/bootloader.bin 0x10000 /tmp/YourBinaryProgramName.bin 0x8000 /tmp/partitions_two_ota.bin'

• Start a Git Bash session and run a shell script for the deploy_OTA.sh file

• Your directory should have the following files in order to properly flash the ESP32

BTW, you can find the deploy_OTA, sh, bootloader.bin and the partitions_two_ota.bin files from the starter directory under the esp32-arduino-ota project when you follow the instructions at Program Over the Air on ESP32 MATRIX Voice w/ Arduino IDE.

git clone https://github.com/matrix-io/esp32-arduino-ota

The Voice Assistant

In order to create a dynamic multiple instrument device, we needed a way to change the instrument current playing. A spoken command creates the most versatile way to accomplish this.

SNIPS Is an AI voice platform for connected devices that animates product interactions with customizable voice experiences.

Start by checking the following project MATRIX Voice and MATRIX Creator Running Snips.aito setup your SNIPS environment.

NOTE: Be aware that the /etc/asound.conf file might need some tweaks in order for the SNIPS processes runs properly. We ended up using the RPi embedded sound chip for sound playing and the Matrix Voice for recording. Also make sure to check the rate value to 16000 for sampling as this is the default for the Matrix to work (please correct us if we are wrong on this).

Once you have your system up and running on SNIPS, we replicated the following project Iron Man Arc Reactor with MATRIX Device and Snips.ai.This give us the steps to create an account on SNIPS and have a first glance on how to create an App.

Ok, we then create our App call MusicGloves, we wanted to keep it private from the time being and set it as unpublished.

We created eight intents, six for musical instruments, one for a mix version of three instruments playing simultaneously (MixOne) and one command to exit the program.

Each intent have one Slot and we create three possible activation sentences: Play Instrument Name, Select Instrument Name and Instrument Name, except for Exit.

Once the Slot is created make sure you press "SAVE", so the system can start training and create the voice assistant.

One thing that was not intuitive was the link of the training example to the slot. You have to select the text of the keyword to associate with the slot and then hover over it. It will display then a popup menu to select the slot. We tried with other versions of our assistant with multiple slots, but we have some issues when receiving the callback function and parsing the JSON message on python. So we decided to create separate intents per instrument.

Now you can deploy your assistant. Follow step 7 of the Iron Man Arc Reactor with MATRIX Device and Snips.ai.

Test the assistant by running

sam watch

and say the hotword "Hey Snips", you will see the process results from the RPi once it recognizes the word.

The Notes

This section explains the second task, the voice command recognition with SNIPS, instrument management and play.

The configuration of Python and SNIPS had a couple of bumps. First we follow the instructions at Listening to intents over MQTT using Python, but it had an issue when trying to install paho-mqtt over Stretch. We found a solution Cannot install paho-mqtt in Python 3.xby running the following commands:

sudo apt-get install mosquitto 
sudo apt-get install mosquitto-clients 
sudo python3 -m pip install paho-mqtt

Once we have our system set, we start by importing our libraries

import paho.mqtt.client as mqtt
from time import sleep
from math import pi, sin
import pygame
import time
import serial
import re
import json

Notice that we will be using pygame to play the instrument sounds, the serial library to communicate with the ESP32 thru the UART port and the mqtt client to link with the SNIPS server.

We initialize our intents to subscribe to the mqtt client. Notice we left by mistake the ArcReactor intent from IronMan project, oops!.

#Snips credentials, make sure you set your Username
snipsUserName = "YourUserName"
Reactor = 'hermes/intent/'+snipsUserName+':ArcReactor'
ViolinMus = 'hermes/intent/'+snipsUserName+':Violin'
CelloMus = 'hermes/intent/'+snipsUserName+':Cello'
PianoMus = 'hermes/intent/'+snipsUserName+':Piano'
FluteMus = 'hermes/intent/'+snipsUserName+':Flute'
TimpaniMus = 'hermes/intent/'+snipsUserName+':Timpani'
ExitMus = 'hermes/intent/'+snipsUserName+':Exit'
MixOneMus = 'hermes/intent/'+snipsUserName+':MixOne'
ChorusMus = 'hermes/intent/'+snipsUserName+':Chorus'

We initialize the mqtt client, set the callback functions for connection and message receive, try to connect to the host and start a new thread running the mqtt messages. Notice that we have the client.loop_forever() function commented. We started with this function but, it takes control of the program and does not let go, therefore code after the function call never executes. We needed something that could run in the background meanwhile we could collect the data coming from the ESP32 script.

#Initialize the mqqt engine
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
#Connect to mqtt service
client.connect(HOST, PORT, 60)
#client.loop_forever()
client.loop_start()

The on_connect callback function from the mqtt service subscribe to the intents after the connection is established.

# Subscribe to the programmed intents
def on_connect(client, userdata, flags, rc):
   print("Connected to {0} with result code {1}".format(HOST, rc))
   # Subscribe to the hotword detected topic
   client.subscribe("hermes/hotword/default/detected")
   # Subscribe to intent topic
   client.subscribe(ViolinMus)
   client.subscribe(CelloMus)
   client.subscribe(PianoMus)
   client.subscribe(FluteMus)
   client.subscribe(ExitMus)
   client.subscribe(TimpaniMus)
   client.subscribe(MixOneMus)
   client.subscribe(ChorusMus)

And the callback function for the mqtt messages sent from SNIPS

# Snips callback function with the detected intent
def on_message(client, userdata, msg):
   global activeInstrument
   if msg.topic == 'hermes/hotword/default/detected':
      print("Hotword detected!")
   elif msg.topic == ViolinMus:
      activeInstrument = ViolinIns
      print("Violin detected!")
   elif msg.topic == CelloMus:
      activeInstrument = CelloIns
      print("Cello detected!")
   elif msg.topic == PianoMus:
      activeInstrument = PianoIns
      print("Piano detected!")
   elif msg.topic == FluteMus:
      activeInstrument = FluteIns
      print("Flute detected!")
   elif msg.topic == TimpaniMus:
      activeInstrument = TimpaniIns
      print("Timpani detected!")
   elif msg.topic == ExitMus:
      activeInstrument = ExitIns
      print("Exit detected!")
   elif msg.topic == MixOneMus:
      activeInstrument = MixOneIns
      print("Mix One detected!")
   elif msg.topic == ChorusMus:
      activeInstrument = ChorusIns
      print("Chorus detected!")

We initialize our pygame sound engine with:

# Initialize the pygame sound engine
pygame.mixer.init()
# Set the number of channels to allow multiple simultaneous sounds
pygame.mixer.set_num_channels(15)

And request 15 Channels to play our instrument sounds simultaneously.

We took a logistic approach to handle the resources as easy as possible, by creating an array of file paths and an array that would create the sound objects so we would not need to load them every time we want to play. A snip of the chorus object is shown below. Notice that the chsndObj array contains all the sound objects for the chorus after loading the corresponding file.

chCEGmfPath = ['/home/pi/MusicOrch/Chorus/chorus-female-c5-PB-loop.wav',
            '/home/pi/MusicOrch/Chorus/chorus-female-e5-PB-loop.wav',
            '/home/pi/MusicOrch/Chorus/chorus-female-g5-PB-loop.wav']
# Create an array of Sound objects for each instrumment. This is the array for chorus.
# Chorus
chsndObj = [pygame.mixer.Sound(chADFmfPath[0]), pygame.mixer.Sound(chADFmfPath[1]),
         pygame.mixer.Sound(chADFmfPath[2]),
         pygame.mixer.Sound(chCEGMmfPath[0]), pygame.mixer.Sound(chCEGMmfPath[1]),
         pygame.mixer.Sound(chCEGMmfPath[2]),
         pygame.mixer.Sound(chCEGmfPath[0]), pygame.mixer.Sound(chCEGmfPath[1]),
         pygame.mixer.Sound(chCEGmfPath[2])]

The selectSoundObj and playNote functions takes the task to select the correct sound according to the active instrument and play on the independent channel.

Finally, the serial port initializes with the below parameters

#Initialize the serial port for communication with the Matrix Voice ESP32 

ser = serial.Serial( port='/dev/ttyS0', baudrate = 115200, parity=serial.PARITY_NONE, stopbits=serial.STOPBITS_ONE, bytesize=serial.EIGHTBITS, timeout=5 )

We use the serial object to request a new set of pressure values, read the values and transmit the active selected object when change by a SNIPS intent command.

if (PauseExec == False):
   # Set read command to ESP32
   ser.write(b'1')
   time.sleep(0.1)
   # Read a line of data
   x = ser.readline()
   # print(x)
   # Convert the binary data into string
   strdecode = x.decode('utf-8')
   # split the data according to the divider
   values = strdecode.split("|")
   # print(values)
   # Play the data
   playNote(activeInstrument, int(values[0]), int(values[1]), int(values[2]))
   time.sleep(0.05)
# If instrument change let know the ESP32 task
if (currIns != activeInstrument):
   print(activeInstrument)
   currIns = activeInstrument
   if (activeInstrument == ViolinIns):
      ser.write(b'A')

Notice we have a PauseExec variable working as a semaphore to synchronize the different events in the program. We used this variable at some point when trying to catch up with a free data flow of the pressure sensors from the ESP32 script, but it did not work that well.

This program controls the flow of data, requesting a new package when it is ready to process it, after playing the values of the previous read package. The data received is in binary format and we convert it to string and split the values on each pressure sensor.

In case the value of the ActiveInstrument is updated by receiving a verbal command from the SNIPS client, the program sends the corresponding code to the ESP32 in order to change the LEDs color.

Timing Is Everything Development (TIED)

It is important to consider the synchronization of events when dealing with a real time stream of data. Before we found the correct process sequence (above), we tried different schemes.

Setting the ESP32 Arduino script to transmit on every cycle a package of the pressure values read, literally clog the UART buffer and the python program was having a difficult time catching up only on the data being received. The SNIPS commands were lacking response and no sound was being played on a timely fashion.

We then try sending a read command to the ESP32 which in turn would read the values at that time, but the readline function would come with half created packages with sensor values missing.

We are still having some crashes in the serial reading part, that is why we implement the try/except case, so we could reset the serial port and restart the communication.

Finding the right synchronization process is complicated but needed in our IOT universe.

Having Musical Fun

Finally, after putting together all the pieces we have our new musical instrument (device). We use a glove with double side tape attached to the pressure sensors and to the plastic ball, which serves are a rigid surface for the sensors to press on.

Things we notice

This project was initially started as using the Matrix Voice ESP32 as a standalone device, without the need of a RPi board.

We wanted to use the ESPNOW protocol to transmit the sensor data to a remote server which would take the responsibility to play the selected instrument. The Matrix Voice ESP32 would be running the SNIPS client and selecting the active instrument, which would be passed as part of the stream package to the remote server.

Trying the ESPNOW sample programs from the ESP32 dev Module worked as a charm.

On a separate script setting the pressure sensor and reading its values worked great.

The problem started when we put together both. For some reason we do not know yet, once the WiFi.mode(WIFI_STA) function was called, the analog reading of the pins stop working. The script start reading random values on each of the ESP32 defined pins in the expansion port.

This took us to drop ESPNOW and create a single device. Then, we have the issue of UART support on the RPi. Since, the first time we make SNIPS worked was with a node.js sample, we search for a library that would support UART on node.js. Needless to say it did not work, oh there are libraries out there for UART and node.js, but we were running out of time.

Finally, we decide to go with python and lucky we found a way to interface with SNIPS.

It has being a long trip. Oh did we mentioned we also look into reprogramming the FPGA to get access to the I2C or the UART or the expansion GPIO. Definitely, a challenge to struck, but not this time. :)

Future work

Add more pressure sensors, figure out the ESPNOW issue and allow for another device to take control of the playing task, and make the device run independently from the RPi board.

Increase the number of intents. We will add an intent like "SELECT VIOLIN and FLUTE" or "SELECT ALL STRINGS" or "INCREASE ONE OCTAVE" or "CHANGE SCALE" or "SENSOR ONE VIOLIN and SENSOR TWO CHORUS", etc. So many possibilities.

We had a blast working on this project. Hope this inspires more projects on music instruments. Thanks for Reading.