Voice Controller in Game

1.Project Planning

Sound plays an important role in human life, we can communicate with each other through language, and current technology has made it possible to give instructions to machines through sound. Through machine learning technology, we can identify and annotate keywords in natural language, which can help people with disabilities control games or devices through voice, bringing convenience to their lives. Google has built a coral dev series of development boards specifically for machine learning, with TPU and other hardware to accelerate the machine learning process, and we plan to use the coral dev mini development board and machine learning model to realize the function of speech recognition control.

2.Part Sourcing

To achieve this goal, we used the google coral dev mini development board as the platform, the microphone on the board as the audio input, the development language was python, the pycoral and pyaudio libraries were used, and the machine learning and classification were tensorflowlite, thanks to Kevin Kilgour, Duc Dung Nguyen, and Sarah Amsellem from google research, Dominik Roblek, Michael Tyka, gave the Coral Keyphrase Detector model, which can be used to annotate speech input.

1） Coral dev mini board

2） Tensorflowlite

3） Pyaudio

4）Coral Keyphrase Detector

https://github.com/google-coral/project-keyword-spotter

3. Building

The main body of the circuit is the coral dev mini development board, using USB connect with the PC, according to the documentation, we can use the mendel tool mdt remote login and operate the development board, the coral development board has been pre-installed with pycoral, only require to install pyaudio, we can install pyaudio by pip3. We use Coral keyphrase Detector model. If you need customize keywords, you need to collect speech and label it, retrain the model, you can refer to TinyML's tutorial on the topic.

The entire circuit is shown in the figure.

4.Coding

We use the model of Coral Keyphrase Detector, MEL features, Audio_recorder code, and we revise the main program, change Snake pygame to detecting speech keywords only.

# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import argparse
import json
import os
from random import randint
from threading import Thread
import time
import model
import queue
import pygame

class Controler(object):
    def __init__(self, q):
        self._q = q
    def callback(self, command):
        self._q.put(command)
class App:
    def __init__(self):
        self._running = True
        pass
    def spotter(self, args):
        interpreter = model.make_interpreter(args.model_file)
        interpreter.allocate_tensors()
        mic = args.mic if args.mic is None else int(args.mic)
        model.classify_audio(mic, interpreter,
        labels_file="config/labels_gc2.raw.txt",
        commands_file="config/commands_v2_snake.txt",
        dectection_callback=self._controler.callback,
        sample_rate_hz=int(args.sample_rate_hz),
        num_frames_hop=int(args.num_frames_hop))
    def on_execute(self, args):
        q = model.get_queue()
        self._controler = Controler(q)
        if not args.debug_keyboard:
            t = Thread(target=self.spotter, args=(args,))
            t.daemon = True
            t.start()
        item = -1
        while self._running:
            if args.debug_keyboard:
                keys = pygame.key.get_pressed()
            else:
                try:
                    new_item = q.get(True, 0.1)
                except queue.Empty:
                    new_item = None
                if new_item is not None:
                    item = new_item
            if (args.debug_keyboard and keys[pygame.K_ESCAPE]) or item == "stop":
                self._running = False
            if (args.debug_keyboard and keys[pygame.K_SPACE]) or item == "go":
                self.game.start()
            if self.game.started():
                if (args.debug_keyboard and keys[pygame.K_RIGHT]) or item == "right":
                    self.game.player.move_right()
                if (args.debug_keyboard and keys[pygame.K_LEFT]) or item == "left":
                    self.game.player.move_left()
                if (args.debug_keyboard and keys[pygame.K_UP]) or item == "up":
                    self.game.player.move_up()
                if (args.debug_keyboard and keys[pygame.K_DOWN]) or item == "down":
                    self.game.player.move_down()
                time.sleep(0.05)
if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument(
        '--debug_keyboard',
        help='Use the keyboard to control the game.',
        action='store_true',
        default=False)
    model.add_model_flags(parser)
    args = parser.parse_args()
    the_app = App()
    the_app.on_execute(args)