Gesture Based Media Control using Nano 33 BLE SENSE

Introduction

The project will be using the internal APDS9960 Gesture Sensor to read gesture data and use the HID features to play/pause, mute/unmute, and change music tracks playing on your laptop.

The datasheet for the Arduino Nano 33 BLE states that the device is HID compatible but the existing keyboard and mouse libraries by Arduino are not compatible with the Nano 33 BLE (as of now).

The Nano 33 BLE boards have their own “USBHID” library that uses the Mbed OS HID functionality. Unfortunately these libraries do not come with examples as of now, so I hope this tutorial/project will help you get started.

Link for USBHID library

Getting Started with HID

NOTE:When sketches with HID functionality are uploaded into the board,you cannot use serial communication and also hence cannot get the Arduino IDE to upload or talk to the board again.

You can get the board back into bootloader mode by pressing on the reset button twice in quick succession and then upload your new sketches. If the reset is done correctly, you should see the “L” LED pulsing to indicate it’s in bootloader mode. If you don’t see the LED pulsing, try the double reset again until you get the timing right. Once the board is in bootloader mode, you will see a port and be able to upload again. Note that the port number may change when it’s in bootloader mode, so make sure to select the correct port from the Arduino IDE’s Tools > Port menu before uploading.

While the USBHID supports both mouse and keyboard interfaces, the following tutorial will only focus on the keyboard implementation.

This is an example code from the library files, note that the wait()function was removed from 1.5.1, use delay() Instead.

#include "mbed.h"
#include "USBKeyboard.h"
USBKeyboard key;
int main(void)
{
 while (1) {
     key.printf("Hello World\r\n");
     wait(1);
 }
} 

It might be very obvious from the start that this isn't in our usual Arduino code format with the setup() and loop().But it does work on the Nano 33 BLE without any errors.

This can be re-written into the more familiar format with setup() and loop() blocks as follows

#include "mbed.h"
#include "USBKeyboard.h"
USBKeyboard key;
void setup() {
   key.printf("Hello World\r\n");
}
void loop() {
}
int main(void)
{
 delay(100); // for the keyboard to have time to be recognized
 setup();
 while(1) loop();
}

Combining HID and Gesture Sensor

If you have not worked with the APDS9960 before please go through the inbuilt examples in the IDE for clarity.

While the above code does work on its own, I was unable to get the APDS9960 to work with this code.

Gesture directions forAPDS9960are as follows:

• UP: from USB connector towards antenna

• DOWN: from antenna towards USB connector

• LEFT: from analog pins side towards digital pins side

• RIGHT: from digital pins side towards analog pins side

Therefore following implementation of the APDS9960 and the USBHID does not work.

#include "mbed.h"
#include "USBKeyboard.h"
#include <Arduino_APDS9960.h>
USBKeyboard key;
void setup() {
   pinMode(13,OUTPUT);
   key.printf("Hello World\r\n");
   if (!APDS.begin()) {
     digitalWrite(13,HIGH);
  }
}
void loop() {
 if (APDS.gestureAvailable()) {
   // a gesture was detected, read and print to serial monitor
   int gesture = APDS.readGesture();
   switch (gesture) {
     case GESTURE_UP:
       key.printf("UP\r\n");
       break;
     case GESTURE_DOWN:
       key.printf("DOWN\r\n");
       break;
     case GESTURE_LEFT:
       key.printf("LEFT\r\n");
       break;
     case GESTURE_RIGHT:
       key.printf("RIGHT\r\n");
       break;
     default:
       // ignore
       break;
   }
 }
}
int main(void)
{
 delay(100); // for the keyboard to have time to be recognized
 setup();
 while(1) loop();
}

I assume this is because of the fact that the Arduino core actually has a hidden main function that internally calls the setup() and loop(), by redefining the main() a few initializations which might be important for the working of the APDS9960 might get skipped which is causing the sensor to not work.

I dug into the Arduino Core files to see what the hidden main function calls, and replaced the main() I defined with the hidden main() function.

I got everything to finally work with the following code:

/* 
Author: Bandi Jai Krishna
Date: 25/06/2021
Summary
The following sketch is for the Arduino Nano 33 BLE SENSE board, it uses the internal gesture sensor(APDS99690) for identifying gestures and then sends appropriate keyboard commands using the USBHID features of the board to control music.
Serial cannot be used for debugging, hence the LED_BUILTIN is being used, if the code runs successfully you should see the LED_BUILTIN stay lit up.
*/
#include <Arduino.h>
#include "mbed.h"
#include "USBKeyboard.h"
#include <Arduino_APDS9960.h>
USBKeyboard key;
// Declared weak in Arduino.h to allow user redefinitions.
int atexit(void (* /*func*/ )()) {
 return 0;
}
// Weak empty variant initialization function.
// May be redefined by variant files.
void initVariant() __attribute__((weak));
void initVariant() { }
void setupUSB() __attribute__((weak));
void setupUSB() { }
void setup() {
 pinMode(13, OUTPUT);
 digitalWrite(13, HIGH);
 if (!APDS.begin()) { 
   digitalWrite(13, LOW);
 }
}
void loop() {
 if (APDS.gestureAvailable()) {
   int gesture = APDS.readGesture();
   switch (gesture) {
     case GESTURE_UP:
       key.media_control(KEY_PLAY_PAUSE);
       break;
     case GESTURE_DOWN:
       key.media_control(KEY_MUTE);
       break;
     case GESTURE_LEFT:
       key.media_control(KEY_PREVIOUS_TRACK);
       break;
     case GESTURE_RIGHT:
       key.media_control(KEY_NEXT_TRACK);
       break;
     default:
       break;
   }
 }
}
int main(void)
{
 init();
 initVariant();
#if defined(USBCON)
 USBDevice.attach();
#endif
 setup();
 for (;;) {
   loop();
   if (serialEventRun) serialEventRun();
 }
 return 0;
}

Results - Gesture Based Music Control

A few other examples for using the USBKeyboard

#include "mbed.h"
#include "USBKeyboard.h"
#include <Arduino_APDS9960.h>
USBKeyboard key;
/*
Media Keys
  KEY_NEXT_TRACK,     //next Track Button
  KEY_PREVIOUS_TRACK, //Previous track Button
  KEY_STOP,           //Stop Button
  KEY_PLAY_PAUSE,     //Play/Pause Button
  KEY_MUTE,           //Mute Button
  KEY_VOLUME_UP,      //Volume Up Button
  KEY_VOLUME_DOWN,   //Volume Down Button   
*/
key.media_control(KEY_NEXT_TRACK); //Example usage
/*
**Modifier Keys**
   KEY_CTRL,
   KEY_SHIFT,
   KEY_ALT,
   KEY_LOGO,
   KEY_RCTRL,
   KEY_RSHIFT,
   KEY_RALT,
   KEY_RLOGO,
*/
key.key_code('s',KEY_CTRL); //Example usage for (save) "CTRL + s"
key.printf("Test"); // Example usage for typing a word. 

References

Arduino Nano 33 BLE Sense HID

Please checkout my blog for staying up to date with my work