I've always been a huge fan of Tony Stark aka Iron man. I was inspired to make a smart voice interaction device that could help me to dispense the accessories in the hotel. In today's world, Voice control has numerous applications on accessibility to various tasks.
This project is used to interact with digital devices using voice recognition which enables you to use voice commands to perform actions using the trained models.
With this solution, we can improve the traditional way of dispensing accessories such as soap, paste, brush, towel by the server. These can be automated using a voice AI.
Hardware BuildFirst of all, I would like to thank MATRIX Labs and SNIPS for supporting this project with the amazing MATRIX Voice. I really felt informative using this board and able to achieve some of the complex projects integrated within a PCB. It has an 8 MEMS microphone to pick the voice commands with great accuracy.
You can purchase the MATRIX Voice ESP32 Version (WiFi/BT/MCU) for $65 from MATRIX Store.
Step 1: Getting Started with MATRIX Voice (Content from MATRIX Labs)MATRIX Voice is a development board for building sound driven behaviors and interfaces. MATRIX Voice was built with a mission to give every maker, tinkerer, developer, and student around the world a complete, affordable, and user-friendly tool for simple to complex Internet of Things (IoT) voice app creation. With the ESP32 module, you are able to run the MATRIX Voice standalone, communicate with other devices, and transmit voice data via Bluetooth and/or WiFi.
Components present in this Matrix Voice board are listed below
- FPGA
- Xilinx Spartan 6 XC6SLX9
- Microphone Array
- 8 MEMS MP34DB02 audio sensor digital microphones
- Everloop
- 18 RGBW LEDs
- RAM
- 64 MByte 132MHz SDRAM
- FLASH
- 64Mbit Flash Memory
- ESP32 (MCU w/WiFi/BT)
- Wifi 2.4Ghz 802.11bgn - BT 4.0 LE - Microcontroller Tensilica Xtensa dual-core 32-bit LX6
Hardware: Raspberry Pi B+ or newer, Raspberry Pi Zero
Software: Raspbian, Debian
Step 2: Interfacing with MATRIX Voice (Basic Project)- Power the kit by the default power adapter in the kit or use a 5V-2A DC adapter with a Micro USB connector.
- The next step is installing the Snips assistant into the Raspberry Pi. To save your setup time, the MicroSD card in the kit has been flashed with the fully functional system image, which means the Snips assistant is ready for your trigger words.
- We strongly suggest you study the step by step installation guide by visiting: Iron Man Arc Reactor with MATRIX Device and Snips.ai by the Team MATRIX Labs.
- Trigger the assistant by saying “JARVIS”, and follow the command.
Python script for Rainbow LED sequence
/*
* Everloop rainbow example
*/
/////////////////////////
// INCLUDE STATEMENTS //
///////////////////////
// System calls
#include <unistd.h>
// Input/output streams and functions
#include <iostream>
// Included for sin() function.
#include <cmath>
// Interfaces with Everloop
#include "matrix_hal/everloop.h"
// Holds data for Everloop
#include "matrix_hal/everloop_image.h"
// Communicates with MATRIX device
#include "matrix_hal/matrixio_bus.h"
int main() {
////////////////////
// INITIAL SETUP //
//////////////////
// Create MatrixIOBus object for hardware communication
matrix_hal::MatrixIOBus bus;
// Initialize bus and exit program if error occurs
if (!bus.Init()) return false;
/////////////////
// MAIN SETUP //
///////////////
// Holds the number of LEDs on MATRIX device
int ledCount = bus.MatrixLeds();
// Create EverloopImage object, with size of ledCount
matrix_hal::EverloopImage everloop_image(ledCount);
// Create Everloop object
matrix_hal::Everloop everloop;
// Set everloop to use MatrixIOBus bus
everloop.Setup(&bus);
// Variables used for sine wave rainbow logic
float counter = 0;
const float freq = 0.375;
// 10 sec loop for rainbow effect 250*40000 microsec = 10 sec
for (int i = 0; i <= 250; i++) {
// For each led in everloop_image.leds, set led value
for (matrix_hal::LedValue &led : everloop_image.leds) {
// Sine waves 120 degrees out of phase for rainbow
led.red =
(std::sin(freq * counter + (M_PI / 180 * 240)) * 155 + 100) / 10;
led.green =
(std::sin(freq * counter + (M_PI / 180 * 120)) * 155 + 100) / 10;
led.blue = (std::sin(freq * counter + 0) * 155 + 100) / 10;
// If MATRIX Creator, increment by 0.51
if (ledCount == 35) {
counter = counter + 0.51;
}
// If MATRIX Voice, increment by 1.01
if (ledCount == 18) {
counter = counter + 1.01;
}
}
// Updates the LEDs
everloop.Write(&everloop_image);
// If i is 0 (first run)
if (i == 0) {
// Output everloop status to console
std::cout << "Everloop set to rainbow for 10 seconds." << std::endl;
}
// If i is cleanly divisible by 25
if ((i % 25) == 0) {
std::cout << "Time remaining (s) : " << 10 - (i / 25) << std::endl;
}
// Sleep for 40000 microseconds
usleep(40000);
}
// Updates the Everloop on the MATRIX device
everloop.Write(&everloop_image);
// For each led in everloop_image.leds, set led value to 0
for (matrix_hal::LedValue &led : everloop_image.leds) {
// Turn off Everloop
led.red = 0;
led.green = 0;
led.blue = 0;
led.white = 0;
}
// Updates the Everloop on the MATRIX device
everloop.Write(&everloop_image);
return 0;
}
You can find the video of the example using the MATRIX Voice interaction kit powered by SNIPS.
Tutorial from Team MATRIX Labs helped me to do this project.
Step 3: Interfacing with MATRIX Voice (Project DISPENSER AI)So this is how it works. First, we use the hot word to initiate Snips recognition on the Matrix Voice. It understands the sentence using Snips NLU and determines the intent. Next using the intent, extract the slots respectively. With the action code, we program the AI response with respect to the slots.
- Create an Assistant. In my case, I had created an assistant with a name Ironman and with a hot-word Snips.
- Create a Custom Application or for beginners, use the Application from the library.
- I had created an Application with the name Dispenser
- Create an Intent and start training the AI. This plays a major role in making your AI robust. Train it with all the possible inputs.
- Create a slot to identify the words. Place the slot at the expected phrase of the sentence. For my example, givenword is the slot name and it is highlighted with blue.
- Once it is done, your custom ASR will be ready. Try with the inputs. We can find the response at your right, below the assistant. This gives you the extraction of raw value, intent name and the slot.
- Define slots with more examples as given below
- Finally, Deploy your assistant to experience your personalized AI.
Raspbian is the Linux distribution of choice running on the Raspberry Pi. In this guide, we will be using the Lite version, but the Desktop version (which comes with a graphical environment) can be used as well.
- Download Etcher and install it.
- Connect an SD card reader with the SD card inside.
- Open Etcher and select from your hard drive the Raspberry Pi
.img
or.zip
file you wish to write to the SD card. - Select the SD card you wish to write your image to.
- Review your selections and click 'Flash!' to begin writing data to the SD card
Connect the device to your network
- Enable SSH access by adding empty file
ssh
, again placed at the root of theboot
volume on your SD card. - Insert the SD card into the Raspberry Pi. It will boot in roughly 20 seconds. You should now have SSH access to your Raspberry Pi. By default, its hostname will be raspberrypi.local.
On your computer, open a terminal window and type the following:
ssh pi@raspberrypi.local
The default password is raspberry
Follow the instructions below for allowing your MATRIX Voice to register as a microphone for your Raspberry Pi.
Please do check this community page if you face any issues with the microphone drivers: https://docs.snips.ai
curl https://apt.matrix.one/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.matrix.one/raspbian $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/matrixlabs.list
sudo apt-get update
sudo apt-get upgrade
The next commands will install the MATRIX kernel modules, overriding the stock Raspbian kernel.
sudo apt install matrixio-kernel-modules
A reboot will be required.
sudo reboot
Check If Everything Works:
Your Raspberry Pi should now treat your MATRIX Voice as a regular microphone. You can test this by using the following commands to record and play a 5-second long audio file on your Raspberry Pi.
Be sure to have something connected to the Raspberry Pi's audio output.
arecord recording.wav -f S16_LE -r 16000 -d 5
aplay recording.wav
Be sure to have something connected to the Raspberry Pi's audio output.arecord recording.wav -f S16_LE -r 16000 -d 5aplay recording.wavStep 7: Install the Snips Platform
If you haven't already, install the Sam Command Line Interface on your computer. Open a terminal window, and enter:
sudo npm install -g snips-sam
or
npm install -g snips-sam //Windows
Find your Raspberry Pi on the network by running:
sam devices
Next, establish a connection to the device. You will be prompted for the device username and password (default is pi and raspberry, respectively):
sam connect <HOSTNAME>
We are now ready to install the Snips platform on the Raspberry Pi. Enter the command:
sam init
Sam provides some commands to easily set up and configure hardware components. In this section, we will set up a speaker and a microphone. Depending on your set up, some custom configuration needs to be done.
sam setup audio
Test the speaker
To check the speaker is working, run
sam test speaker
If everything works fine, you should hear on your speaker a voice saying a few words.
Configure the microphone
Type the following command on the SSH to configure the microphone.
sudo nano /etc/snips.toml
Scroll down to where you see [snips-audio-server]
and replace
# mike = "Built-in Microphone"
with the following:
mike = "MATRIXIO-SOUND: - (hw:2,0)"
You can check that your microphone is working:
sam test microphone
Assuming you have a speaker, if you can clearly hear what you just said, you can move on without further microphone configuration.
Once you are satisfied that your hardware setup is working, you are ready to move on to the next step, where you will deploy your first assistant.
Step 9: Install MATRIX Lite JSInstalling the MATRIX Kernel Modules (Raspberry Pi)
Run the following commands in your Raspberry Pi's terminal to add the MATRIX repository & key and update your repository packages.
Add the MATRIX repository and key.
curl https://apt.matrix.one/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.matrix.one/raspbian $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/matrixlabs.list
Update your repository and packages.
sudo apt-get update
sudo apt-get upgrade
Install the MATRIX HAL packages.
sudo apt-get install matrixio-creator-init libmatrixio-creator-hal libmatrixio-creator-hal-dev
Reboot your device.
sudo reboot
Note:
- MATRIX HAL header files are installed in
/usr/include/matrix_hal
- The compiled MATRIX HAL library file is installed in
/usr/lib/libmatrix_creator_hal.so
- Run the following command in your terminal:
cat /etc/asound.conf
If you have the default audio configuration, the command will output something very close to this:
pcm.!default {
type asym
playback.pcm {
type plug
slave.pcm "hw:0,0"
}
capture.pcm {
type plug
slave.pcm "hw:2,0"
}
}
Step 10: Importing AI model from the consoleOnce you have created your assistant from the previous steps, log in to the console from the terminal window by running:
sam login
You will be asked to enter your Snips Console credentials.
Next, install the assistant:
sam install assistant
If you have created several assistants in the Console, you will be asked to choose which one to install from a list.
Step 11: Handler & Action codeIn order for the Raspberry Pi to respond, we will have to write some handler code. This is code that is executed when a certain event happens on the Snips platform, such as when intent has been detected.
You have various methods to create Handler depending on the functioning.
Create a folder on your computer on which you want to store your handler code. From the terminal window, change to that directory, and enter:
npm init
This will create a file named package.json
which contains your project's description, dependencies and more. Next, add the mqtt
package dependency:
npm install mqtt --save
Now, create a file called index.js
, and paste the following code, replacing the hostname
variable with the hostname of your Raspberry Pi (if it differs from the default raspberrypi
):
<action - script>
Run this code using Node:
node index.js
You should see a message of a successful connection to the Snips MQTT broker. This code helps us to control the servo motors depending on the Voice Intends received.
As before, perform a voice interaction with your Raspberry Pi with fake Weather demo:
Hey Snips, what is the weather in Chennai
You should see the logs updating:
[Snips Log] Connected to MQTT broker mqtt://raspberrypi.local
[Snips Log] Hotword detected
[Snips Log] Start listening
[Snips Log] Stop listening
[Snips Log] Intent detected: {"sessionId":"c84b5aa5-3f14-4218-975e-8872b9217933",
"customData":null,"siteId": "default",
"input":"what is the weather in chennai",
"intent":{"intentName":"searchWeatherForecast","probability":0.73845243},
"slots":[{"rawValue":"chennai","value":{"kind":"Custom","value":"Chennai"},"range":{"start":44,"end":53},"entity":"locality","slotName":"forecast_locality"}]}
Troubleshooting- Delete the previous assistant if there is one.
$ sudo rm -rf /usr/share/snips/assistant/
If
there is any issue with the Audio Server, kindly following this issue
https://github.com/snipsco/snips-issues/issues/203
- To change the directory to the Assistant
cd /var/lib/snips/skills/
- To Clone the content of the repo to local
git clone <git_link>
- To run the setup.sh script
./setup.sh
- Activate virtual environment
source venv/bin/activate
- Run the action code
./action-<filename>.py
- To give permission to the Python Script
chmod +x action-<filename>.py
Step 12: ConnectionsServo motors are used to dispense the required accessories.
The Servo motors are connected to the GPIO Pins of the Matrix Voice.
The Connections are shown below.
Each servo is connected to a slider mechanism which will drop the accessories in which the voice intend is invoked.
Step 13: EnclosureI had planned to use an Acrylic enclosure which was to be laser cut for this project. Unfortunately, the outer body of the dispenser is being delayed by the local vendor due to internal issues. And I have no hope that I will receive the enclosure for this project in time.
Anyways I have added the files necessary for this project.
You can find the data which is monitored using Snips-watch
Now it's showtime. I started making with Simple IronMan's Reactor. Have a look at it's Functioning.
The Dispenser AI has been programmed for a limited item. This is extendable with little modifications. This helps the customers to get the bathroom accessories faster than the traditional method.
Video will be updated soon
Kind Attention:
This project can be further expanded by making minor tweaks in the action code.
You can find the Action scripts on my Github Repository attached.
Link to My application: https://console.snips.ai/store/en/skill_7wAyA2w7dpD
The entire setup costs about $100 which can be purchased from MATRIX Store and this solution is very cheap and effective when compared to the traditional methods.
.
.
.
Vote of Thanks:
After this contest, I will update the project with more improvements.
Many thanks to Hackster for allowing me and all the users of this great platform to participate.
Thank you for carrying out these activities that encourage young people and teenagers to create and invent.
Give a thumbs up if it really helped you and do follow my channel for interesting projects. :)
Share this video if you like.
Happy to have you subscribed: https://www.youtube.com/channel/UCks-9JSnVb22dlqtMgPjrlg/videos
Thanks for reading!
Comments