Published October 8, 2019 © GPL3+

Smart AI Gloves

AI-based hand gloves, gives you a super power to control any devices with motions.

IntermediateShowcase (no instructions)5 days5,234

Best Use of AI at the Edge: 2nd Place

Bring Intelligence to the Real World

Things used in this project

Hardware components

Avnet SmartEdge Agile Brainium

Raspberry Pi Zero

NXP Rapid IoT Prototyping Kit

Fan Kit, 120 mm Fan

LED Light Bulb, Frosted GLS

RC car

NXP Kinetis KW41Z

KW41Z Thread Nodes

Hand Gloves

Software apps and online services

Brainium AI Studio

Raspberry Pi Raspbian

Hand tools and fabrication machines

Soldering iron (generic)

Story

AI Gloves

It was my old thought of making some gadgets of super heroes like Iron Man. After analyzing the Avnet SmartEdge device and Brainium cloud, got some idea to make such kind of wearable gloves. Controlling devices in gesture manner.

I received the Avnet kit, the first thing I explored about the Branium cloud and basic features. I have done very basic level project here.

Followed by my basic demo, I started this project activity. I listed out the motions required in my project. Then with AI Studio I reordered many motions and created modules. Started testing with mobile app initially. Then later I started using Brainium app in Raspberry Pi.

Motions Set

Brainium APP on Raspberry pi zero

Also integrated the Rapid IoT kit for thread networking and display. Its talks with all the thread end nodes and controls it.

Rapid IoT Kit

The Main Features of AI Gloves:

1. Unlocking by drawing pattern

2. Light ON/OFF

3. Fan speed control

4. Gesture car control

5. Driver style analyzing in real-time

Blocks:

Python is used to detect the motions/alerts from Brainum cloud using MQTT services. Also it process the motion types then send commands to Rapid IoT kit via UART. Rapid IoT takes the action via Thread. Due to cloud based processing, it will take 3 to 5sec to update the results. But still Brainuim performed well overall.

Avnet SmartEdge device has other inbuilt sensors like magnetic, PIR, accelerator sensors which helps in the driver style analyzing, where the driving information can be used to automate or learning purposes in autonomous car industries.

There are some more demos I have planned in future with this all sensors.

Python MQTT

The working demo is here:

Code

void APP_Init
(
    void
)
{
    /* Initialize pointer to application task message queue */
    mpAppThreadMsgQueue = &appThreadMsgQueue;

    /* Initialize main thread message queue */
    ListInit(&appThreadMsgQueue.msgQueue,APP_MSG_QUEUE_SIZE);

    /* Set default device mode/state */
    APP_SetState(gThrDefaultInstanceId_c, gDeviceState_FactoryDefault_c);
    APP_SetMode(gThrDefaultInstanceId_c, gDeviceMode_Configuration_c);

    /* Initialize keyboard handler */
    pfAppKeyboardHandler = App_HandleKeyboard;

    /* Use one instance ID for application */
    mThrInstanceId = gThrDefaultInstanceId_c;

#if THR_ENABLE_EVENT_MONITORING
    /* Initialize event monitoring */
    APP_InitEventMonitor(mThrInstanceId);
#endif

    if(gThrStatus_Success_c == THR_StartInstance(mThrInstanceId, pStackCfg[0]))
    {
        /* Initialize CoAP demo */
        APP_InitCoapDemo();

#if USE_TEMPERATURE_SENSOR
        /* Initialize Temperature sensor/ADC module*/
        APP_InitADC(ADC_0);
#endif

#if THREAD_USE_THCI && THR_ENABLE_MGMT_DIAGNOSTICS
        (void)MgmtDiagnostic_RegisterAppCb(THCI_MgmtDiagnosticAppCb);
#endif

#if THREAD_USE_SHELL && SOCK_DEMO
        /* Initialize use sockets - used from shell */
        APP_InitUserSockets(mpAppThreadMsgQueue);
#endif

#if APP_AUTOSTART
       if(!THR_GetAttr_IsDevConnected(mThrInstanceId))
       {
          mJoiningIsAppInitiated = TRUE;

          if(THR_NwkJoin(mThrInstanceId, THR_APP_JOIN_DISCOVERY_METHOD) != gThrStatus_Success_c)
          {
              /* User can treat join failure according to their application */
          }
       }
#endif
    }
}

/*!*************************************************************************************************
\fn     void App_Handler(void)
\brief  Application Handler. In this configuration is called on the task with the lowest priority
***************************************************************************************************/
void APP_Handler
(
    void
)
{
    bool_t handleMsg = TRUE;

    while(handleMsg == TRUE)
    {
        handleMsg = NWKU_MsgHandler(&appThreadMsgQueue);
        /* For BareMetal break the while(1) after 1 run */
        if(!gUseRtos_c && MSG_Pending(&appThreadMsgQueue.msgQueue))
        {
            (void)OSA_EventSet(appThreadMsgQueue.taskEventId, NWKU_GENERIC_MSG_EVENT);
            break;
        }
    }
}

static void APP_ProcessLedCmd
(
    uint8_t *pCommand,
    uint8_t dataLen
)
{
    /* Set mode state */
    APP_SetMode(mThrInstanceId, gDeviceMode_Application_c);

    /* Process command */
    if(FLib_MemCmp(pCommand, "on", 2))
    {
        App_UpdateStateLeds(gDeviceState_AppLedOn_c);
    }
    else if(FLib_MemCmp(pCommand, "off", 3))
    {
        App_UpdateStateLeds(gDeviceState_AppLedOff_c);
    }
    else if(FLib_MemCmp(pCommand, "toggle", 6))
    {
        App_UpdateStateLeds(gDeviceState_AppLedToggle_c);
    }
    else if(FLib_MemCmp(pCommand, "flash", 5))
    {
        App_UpdateStateLeds(gDeviceState_AppLedFlash_c);
    }
    else if(FLib_MemCmp(pCommand, "rgb", 3))
    {
        char* p = (char *)pCommand + strlen("rgb");
        uint8_t redValue = 0, greenValue = 0, blueValue = 0;
        appDeviceState_t appState = gDeviceState_AppLedRgb_c;

        dataLen -= strlen("rgb");

        while(dataLen != 0)
        {
            if(*p == 'r')
            {
                p++;
                dataLen--;
                redValue = NWKU_atoi(p);
            }

            if(*p == 'g')
            {
                p++;
                dataLen--;
                greenValue = NWKU_atoi(p);
            }

            if(*p == 'b')
            {
                p++;
                dataLen--;
                blueValue = NWKU_atoi(p);
            }

            dataLen--;
            p++;
        }

        /* Update RGB values */
#if gLedRgbEnabled_d
        Led_UpdateRgbState(redValue, greenValue, blueValue);
#else
        appState = gDeviceState_AppLedOff_c;
        if(redValue || greenValue || blueValue)
        {
            appState = gDeviceState_AppLedOn_c;
        }
#endif
        App_UpdateStateLeds(appState);              
    }
    else if(FLib_MemCmp(pCommand, "color wheel", 11))
    {
#if gLedRgbEnabled_d
        App_UpdateStateLeds(gDeviceState_AppLedColorWheel_c);
#else
        App_UpdateStateLeds(gDeviceState_AppLedFlash_c);
#endif
    }
}


static void APP_CoapDriveCb
(
coapSessionStatus_t sessionStatus,
void *pData,
coapSession_t *pSession,
uint32_t dataLen
)
{

	if (gCoapNonConfirmable_c == pSession->msgType)
	{


	if(FLib_MemCmp(pData, "left", 4))
	{
		shell_write("Turn LEFT");
		Drive_Left();
	}

	if(FLib_MemCmp(pData, "right", 4))
	{
		shell_write("Turn RIGHT");
		Drive_Right();
	}


	if(FLib_MemCmp(pData, "front", 5))
	{
		shell_write("Move FRONT");
		Drive_Front();
	}

	if(FLib_MemCmp(pData, "back", 4))
	{
		shell_write("Move BACK");
		Drive_Back();
	}
	if(FLib_MemCmp(pData, "stop", 4))
	{
		shell_write("Stop");
		Drive_Stop();
	}

	if(FLib_MemCmp(pData, "ledon", 5))
	{
		//App_UpdateStateLeds(gDeviceState_AppLedOn_c);
		shell_write("Turn ON LIGHT");
		Drive_LEDON();
	}

	if(FLib_MemCmp(pData, "ledoff", 6))
	{
		//App_UpdateStateLeds(gDeviceState_AppLedOn_c);
		shell_write("Turn OFF LIGHT");
		Drive_LEDOFF();
	}

	shell_writeN(pData, dataLen);
	shell_write("\r\n");
	}
}

#include <raspi.h>
#include "board.h"
#include "fsl_uart.h"

#include "pin_mux.h"
#include "clock_config.h"
#include "uart.h"
#include "backlight.h"
#include "emwin_support.h"
#include "GUI.h"
#include "fsl_ili9341.h"
#include "port_interrupts.h"
#include "images.h"
#include "uidraw.h"
#include "button.h"
#include "thread.h"

/*******************************************************************************
 * Definitions
 ******************************************************************************/
#define BOARD_SW_IRQ PORTE_IRQn
#define BOARD_SW_IRQ_HANDLER PORTE_IRQHandler
#define APP_MAX_COUNT 5
#define APP_TIMER 10
#define APP_LOCK_TIMER 2500
/*******************************************************************************
 * Prototypes
 ******************************************************************************/

/*******************************************************************************
 * Variables
 ******************************************************************************/

uint8_t txbuff[] = "version\r\n";
uint8_t ifconfig[] = "ifconfig\r\n";
uint8_t AIData[4] = {0};
/* Whether the SW button is pressed */
volatile bool g_ButtonPress = false;
volatile uint8_t g_ButtonValue = 0;
volatile uint32_t g_cloud_counter=0;
volatile uint32_t g_lock_counter=0;
volatile uint8_t g_AppIndex = 0;
volatile uint8_t g_AppLocked = 0;
volatile uint8_t g_AppLockDisable = 0;
volatile uint8_t g_AppStatus[5] = {0,0,0,0,0};
volatile uint8_t g_AppValue[5] = {0,0,0,0,0};
extern uint8_t RxRpiBuffer[8];
extern volatile uint8_t rxRpiBlockReceived;
extern volatile uint8_t rxRpiDataCounter;
/* Functions required for emwin / Display */
uint32_t DSPI2_GetFreq(void)
{
    return CLOCK_GetBusClkFreq();
}


void Init_Display()
{
    Display_Connect(); /* triggers GUI_Init() */
    Backlight_SetLevel(BLIGHT_LEVEL_HIGH);

    GUI_SetBkColor(GUI_BLACK);
    GUI_SetColor(GUI_WHITE);
}



#define GUI_FONT_TITLE  GUI_Font8x18
#define GUI_FONT_NORMAL GUI_Font8x16

uint8_t Init_Backlight(void)
{
	backlight_fct_t BACKLIGHT_fct;

    BACKLIGHT_fct.connect_hw = Backlight_Connect;
    BACKLIGHT_fct.disconnect_hw = Backlight_Disconnect;
    BACKLIGHT_fct.set_level = Backlight_Set_Level;

    Backlight_Init_Driver(&BACKLIGHT_fct);

    if (Backlight_Init_Hw())
        return 1;

    return 0;
}


/*******************************************************************************
 * Code
 ******************************************************************************/
/*!
 * @brief Main function
 */
int main(void)
{

	BOARD_InitPins();
    BOARD_Boot_Clock_RUN();

    BOARD_INIT_GPIOS();
    BOARD_Init_SPI_2();

    BOARD_Init_RGB_BLIGHT();
    PORT_IRQ_EnablePortAIrq();
    PORT_IRQ_EnablePortEIrq();

    THR_UART_Init();

    RPI_UART_Init();

    Init_Backlight();

    Init_Display();
    GUI_Clear();
    GUI_SetColor(GUI_DARKMAGENTA);
    GUI_SetBkColor(GUI_WHITE);
    //GUI_SetFont(&GUI_Font24B_1);//GUI_Font32_ASCII
    GUI_SetFont(&GUI_Font20B_1);
    GUI_SetPenSize(8);
	GUI_Clear();

	GUI_BMP_Draw(bglogo_176x176, 0, 0);
    App_WaitMsec(3000);
	GUI_BMP_Draw(bg_176x176, 0, 0);
    App_WaitMsec(100);
	draw_apppage(g_AppIndex,1,0);

    while (1)
    {

    		App_WaitMsec(10);

    		g_lock_counter++;
    		if((g_lock_counter>=APP_LOCK_TIMER) && (g_AppLockDisable==0)){
    			g_AppLocked=1;
    			//GUI_DispStringAt(" Locked  ", 40, 140);
    			GUI_BMP_Draw(logo_lock_screen_128x128, 24, 40);
    			Backlight_SetLevel(BLIGHT_LEVEL_OFF);
    			g_lock_counter=0;
    		}

    		if(g_ButtonPress){


				switch(g_ButtonValue){

				case 0:

					break;

				case 1:
					if(!g_AppLocked){
						g_AppIndex++;
					}
					g_AppLocked=0;
					g_lock_counter=0;
	    			Backlight_SetLevel(BLIGHT_LEVEL_HIGH);
					if(g_AppIndex>3){
						g_AppIndex=0;
					}
					if(g_AppIndex==3) g_AppLockDisable=1;
					else g_AppLockDisable=0;

					draw_apppage(g_AppIndex,1,0);
					break;

				case 2:
					if(!g_AppLocked){
						g_lock_counter=0;
						g_AppValue[g_AppIndex]++;
						if(g_AppValue[g_AppIndex]>2)g_AppValue[g_AppIndex]=0;
						draw_textbox(g_AppIndex, g_AppValue[g_AppIndex]);
					}
					break;

				case 3:
					if(!g_AppLocked){
						rxRpiBlockReceived = true;
						rxRpiDataCounter =0;

						RPI_ClearRxBuffer();
						if(g_AppIndex==0)
						Write_Node(0, 1);

						if(g_AppIndex==1)
						Write_Node(1, 2);

						if(g_AppIndex==2){
						Write_CarNode(2, 4);
						App_WaitMsec(2000);
						Write_CarNode(2, 6);
						}
					}
					break;
				case 4:
					if(!g_AppLocked){
						if(g_AppIndex==0)
						Write_Node(0, 0);

						if(g_AppIndex==1)
						Write_Node(1, 0);

						if(g_AppIndex==2){
							Write_CarNode(2, 5);
							App_WaitMsec(2000);
							Write_CarNode(2, 6);

						}
					}
				break;

				}

				g_ButtonValue = 0;
				g_ButtonPress = false;

    		}

				if(rxRpiBlockReceived){

					RPI_ReadData();
					g_cloud_counter=0;

					if((AIData[0]==1) && (g_AppLocked)){
						g_lock_counter=0;

						GUI_DrawLine(23+112, 38+16, 23+64, 38+16);
						App_WaitMsec(150);
						GUI_DrawLine(23+64, 38+16, 23+16, 38+16);
						App_WaitMsec(150);
						GUI_DrawLine(23+16, 38+16, 23+16, 38+64);
						App_WaitMsec(150);
						GUI_DrawLine(23+16, 38+64, 23+16, 38+112);
						App_WaitMsec(150);
						GUI_DrawLine(23+16, 38+112, 23+64, 38+64);
						App_WaitMsec(150);
						GUI_DrawLine(23+64, 38+64, 23+112, 38+16);
						App_WaitMsec(150);
						Backlight_SetLevel(BLIGHT_LEVEL_HIGH);
						App_WaitMsec(150);
						//GUI_DispStringAt("UnLocked ", 40, 140);

						draw_apppage(g_AppIndex,0,0);
						g_AppLocked=0;
					}




					switch(AIData[2] | (g_AppLocked<<7)){

					case 0:

						break;

					case 1:
						g_lock_counter=0;
						//GUI_DispStringAt(" TurnON  ", 40, 140);
						if(g_AppIndex==0)
							Write_Node(0, 1);
						else if(g_AppIndex==1)
							Write_Node(1, 1);
						break;

					case 2:
						g_lock_counter=0;
						//GUI_DispStringAt(" TurnOFF ", 40, 140);
						if(g_AppIndex==0)
							Write_Node(0, 0);
						else if(g_AppIndex==1)
							Write_Node(1, 0);
						break;

					case 3:
						g_lock_counter=0;
						//GUI_DispStringAt(" MoveUP  ", 40, 140);
						if(g_AppIndex==1)
						Write_Node(1, 2);
						break;
					case 4:
						g_lock_counter=0;
						//GUI_DispStringAt(" MoveDwn ", 40, 140);
						if(g_AppIndex==1)
						Write_Node(1, 1);
						break;
					case 5:
						g_lock_counter=0;
						//GUI_DispStringAt(" Left     ", 40, 140);
						Write_CarNode(2, 2);
						App_WaitMsec(2000);
						Write_CarNode(2, 6);
						break;
					case 6:
						g_lock_counter=0;
						//GUI_DispStringAt(" Right   ", 40, 140);
						Write_Node(2, 3);
						App_WaitMsec(2000);
						Write_CarNode(2, 6);
						break;
					case 7:
						g_lock_counter=0;
						//GUI_DispStringAt(" Front    ", 40, 140);
						Write_CarNode(2, 4);
						App_WaitMsec(2000);
						Write_CarNode(2, 6);
						break;
					case 8:
						g_lock_counter=0;
						//GUI_DispStringAt(" Back    ", 40, 140);
						Write_Node(2, 5);
						App_WaitMsec(2000);
						Write_CarNode(2, 6);
						break;
					}


				}


    }
}

import uuid
import paho.mqtt.client as mqtt
import requests
import json
import re

def parsedata(payload):
    global jdata,Motion_Type,Actual_Speed,Actual_Time
    jdata = json.loads(payload)
    Actual_Speed=jdata[0]['speed']
    Actual_Time=(jdata[0]['finishedAt']-jdata[0]['startedAt'])
    print "Time=",Actual_Time 
    print "Speed=",Actual_Speed
    return Motion_Type

def on_connect(client, userdata, flags, rc):
    print('Connected with result code {code}'.format(code=rc))


def on_disconnect(client, userdata, flags, rc):
    print('Disconnected with result code {code}'.format(code=rc))


def on_message(client, userdata, msg):
    #print('Msg received from topic={topic}\n{content}'.format(topic=msg.topic, content=str(msg.payload)))
    rxdata = parsedata(msg.payload)
    print rxdata

def main():
    print "Starting\n"
    client = mqtt.Client(client_id=str(uuid.uuid4()), transport='websockets')
    client.on_connect = on_connect
    client.on_message = on_message
    client.on_disconnect = on_disconnect

    client.tls_set(ca_certs=ca_cert_path)
    client.username_pw_set(mqtt_user_name, mqtt_password)

    client.connect('ns01-wss.brainium.com', 443)
    print "connected"
    client.subscribe(motions_topic)
   

    client.loop_forever()