Team TapLock: Gal Likar, Urh Smrkolj, ziga andrejc, Jakob Jenko

Published May 28, 2021 © GPL3+

TapLock - A bike lock with machine learning

Use the power of machine learning to read your tapping pattern and keep your bike safe from potential thieves.

IntermediateFull instructions provided8 hours4,953

TapLock - A bike lock with machine learning

Things used in this project

Hardware components

Arduino Nano 33 BLE Sense

Android device

Or an iOS device.

Powerbank

USB-A to Micro-USB Cable

Software apps and online services

Punch Through LightBlue® — Bluetooth Low Energy

Hand tools and fabrication machines

3D Printer (generic)

Required for printing the case, that mounts on the bike.

Story

A centuries old problem solved with this years technology - Machine learning vs bike thieves.

The fear of having your bike stolen has for years been something that bicycle owners had to live with. The only solution for this problem was the boring old bicycle chain, that although effective is something that belongs in the past century.

We present to you: TapLock. A lock, that mounts anywhere on you steel beast and protects it from possible thieves - 21st century style.

Machine learning with Edge impulse

The platform that we use to implement ML is Edge impulse. It allows for a very user friendly interaction and allows you to customize many parameters and block types in the ML chain.

For tap detection we used the accelerometer that can be found on the Arduino Nano 33 BLE sense. By default, Edge impulse records the three separate axes for 5000 milliseconds and presents it as a graph of axial acceleration based on time.

What worked best for us, was preprocessing the accelerometer output on the board by averaging the acceleration over all three axes - taps and vibrations caused by them don't really have a direction. We then took the time between two peaks as the time between taps and outputted it as four numbers for a five tap sequence(time in milliseconds between the first and second peak, second and third,...).

An example of time values in a .json sample.

This data was then imported into Edge impulse as a.json file with a window size of 1 ms and 4 axes(4 peak-to-peak times). We used the raw data processing block and Keras neural network learning block. We changes the Neural network parameters a bit and used 500 training cycles with 0.005 learning rate and 0.60 minimum confidence rating. With those settings, we were able to get 96.4% accuracy.

Results of Neural network training.

The different tap sequences(with a morse code equivalent) use were as followed:

A demonstration of tap pattern recognition with debugging.

Bluetooth connection

The work flow of the lock-app bluetooth connection is as follows:

1. Arduino advertises itself openly on Bluetooth, and the app checks if the localname is correct. If it finds a device, which was the last connected device it chooses it as a priority.

2. The application checks its locally saved file, which contains information about the last locked/unlocked status and the map pins.

3. The app reads the current Arduino status(locked/unlocked) and updates it's file.

4. When we wish to change the lock status, the app waits for an acknowledgment message from the lock only then does the status change in the app.

Native app

A demonstration of locking and unlocking.

App was developed in programming language react native. It consist of three screens. The first one allows the user to connect with arduino and lock/unlock device. Devices are represented in different views as scrollable list. User can also check for more detailed informations of device, such as battery level and distance to the device by long pressing on device name.

The second screen is used for maps. When the user locks the device, a marker is created on this screen. We can also create marker manually and store the location with a press on the marker. The last screen is reserved for app setting (sign up, log in). We can navigate between the screens with tapping on navigator on the bottom of the app.

Maps feature incorporated into the app.

3Dprintablebikemount

We have designed a simple 3D printable mount, that securely houses your Arduino Nano and allows you to mount it virtually anywhere on your bike. All you need is a 3D printer and some rubber bands. The mounting points are designed in such a way, that it is possible to use a single rubber band, but if you want extra security, you can double up on them.

A simple yet effective case.

The microcontroller can be mounted anywhere on the frame of the bicycle, and should be able to read any signal you tap on the frame.

Note: At the current stage of the project the solution for powering your Arduino Nano is a powerbank attached to your bicycle connected with a USB cable

The three components are all mounted on the bicycle.

. We recommend using rubber bands and/or duck tape to fix it to the bicycle.

How it all comes together

The entire package works as followed:

1. You connect to your lock.

2. You lock your lock, which also drops a marker on your current position.

3. The 'locked status' LED turns on(a feature, that can later be expanded e.g. connect an actuator to some pins to engage a lock)

4. You can then unlock it via the app or the correct tapping sequence.

5. The 'unlocked status' LED turns on.

As you might have guessed by now, currently, the lock doesn't feature any lock :) We made the groundwork, so it is rather simple to add your own locking mechanism. The only thing you need to change are the pins on which the 'locked' status is outputted and connect that to your mechanism.

The ML part of the project works surprisingly well and the taps are transmitted over the bike frame, so it is possible to tap a few centimeters away from the Arduino. Experiment :)

A video story presentation of all the functions.

Code

/* Edge Impulse Arduino examples
 * Copyright (c) 2021 EdgeImpulse Inc.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 * SOFTWARE.
 */

/* Includes ---------------------------------------------------------------- */
#include <tapcode5.0_inference.h>
#include <Arduino_LSM9DS1.h>
#include <ArduinoBLE.h>


#define interval2           120

/* Constant defines -------------------------------------------------------- */
#define CONVERT_G_TO_MS2    9.80665f

/* Private variables ------------------------------------------------------- */
static bool debug_nn = false; // Set this to true to see e.g. features generated from the raw signal


float x, y, z, a, a_last, da;
int knock,sum,reading;
int i = 0;
unsigned long diffArr[5];

// BLE Battery Service
BLEService predictionService("180F");
//BLEService ledService("19B10000-E8F2-537E-4F6C-D104768A1214"); // BLE LED Service
BLEByteCharacteristic switchCharacteristic("19B10001-E8F2-537E-4F6C-D104768A1214", BLERead | BLEWrite);
// BLE Battery Level Characteristic
//BLEUnsignedCharCharacteristic batteryLevelChar("2A19",  // standard 16-bit characteristic UUID
//    BLERead | BLENotify); // remote clients will be able to get notifications if this characteristic changes

BLEUnsignedCharCharacteristic LockStatusBLE("2A18",  // standard 16-bit characteristic UUID
    BLERead | BLENotify); // remote clients will be able to get notifications if this characteristic changes

int oldBatteryLevel = 0;  // last battery level reading from analog input
long previousMillis = 0;  // last time the battery level was checked, in ms
int encodedPrediction = 0; 

bool lockState = 0;


float features[] = {1, 1, 1, 1};

/**
 * @brief      Copy raw feature data in out_ptr
 *             Function called by inference library
 *
 * @param[in]  offset   The offset
 * @param[in]  length   The length
 * @param      out_ptr  The out pointer
 *
 * @return     0
 */
int raw_feature_get_data(size_t offset, size_t length, float *out_ptr) {
    memcpy(out_ptr, features + offset, length * sizeof(float));
    return 0;
}


/**
 * @brief      Arduino setup function
 */
void setup()
{   
  
    pinMode(22, OUTPUT);
    pinMode(23, OUTPUT);
    pinMode(24, OUTPUT);
    pinMode(25, OUTPUT);
    digitalWrite(22, LOW);
    digitalWrite(23, LOW);
    digitalWrite(24, LOW);
    digitalWrite(25, LOW);

    // put your setup code here, to run once:
    Serial.begin(115200);
    while (!Serial);
    Serial.println("Edge Impulse Inferencing Demo");

    if (!IMU.begin()) {
        ei_printf("Failed to initialize IMU!\r\n");
    }
    else {
        ei_printf("IMU initialized\r\n");
    }
    // begin initialization
    if (!BLE.begin()) {
      Serial.println("starting BLE failed!");
  
      while (1);
    }

    BLE.setLocalName("tap-lock");
    BLE.setAdvertisedService(predictionService); // add the service UUID
    
//    predictionService.addCharacteristic(batteryLevelChar); // add the battery level characteristic
//    batteryLevelChar.writeValue(42); // set initial value for this characteristic

    predictionService.addCharacteristic(LockStatusBLE); // add the battery level characteristic
    LockStatusBLE.writeValue(1); // set initial value for this characteristic
    
    predictionService.addCharacteristic(switchCharacteristic);

    BLE.addService(predictionService); // Add the battery service

    switchCharacteristic.writeValue(0);
    // start advertising
    BLE.advertise();
    
    Serial.println("Bluetooth device active, waiting for connections...");
}

/**
 * @brief      Arduino main function
 */
void loop(){
 
    lock();
    // wait for a BLE central
    BLEDevice central = BLE.central();   
    static unsigned long last_interval_ms = 0;
    static unsigned long last_interval2 = 0;
    static unsigned long last_knock = 0;
    static unsigned long diff = 0;
    
    // if a central is connected to the peripheral:
    //central
    if (central) {
      Serial.print("Connected to central: ");
      // print the central's BT address:
      Serial.println(central.address());
      digitalWrite(LED_BUILTIN, HIGH);

      
    while (central.connected()) {   
        // read to the end of the buffe
        float x, y, z, a;
        last_interval_ms = millis();
        IMU.readAcceleration(x, y, z);
        a = ((x+y+z)/3)*100*5;
        da = a - a_last;  
        //abs(da) > 7 tukej se lahko igramo s tresholdom veja cifra manj obutiljiv
        if(abs(da) > 7 && (millis() > last_interval2 + interval2)){
          //Serial.println(millis()-last_interval2);
          //calculate time passed since last "tap"
          diff = millis()-last_interval2;
          //if more that 3 seconds have passed reset tap counter to 0
          if(diff > 2000){
              i = 0;
              diff = 0;
          }
//          if(diff > 1500){
//            diff = 1500;
//          }
          //knock = diff;
          last_interval2 = millis();
          // UNCOMMMENT FOR ****** DEBUG ****
          //has to be commented while recording
          Serial.print("DEBUG dt: ");
          Serial.println(diff);
          
          
          // od 0 do 3 gre i tko nardimo array ki ga uporabmo
          //difArr rabi algoritem
          diffArr[i]=diff;

          if (i >= 4){
            i = 0;
            sum = diffArr[1]+diffArr[2]+diffArr[3]+diffArr[4];
            if(diffArr[0] == 0 && sum > 1000){
              
              //printJsonStart();
              for(int n=1; n <= 4; n++){
  
                  features[n-1] = diffArr[n];
                  Serial.println(features[n-1]);
                  
                if(n <=3 && n != 0){
                  Serial.print(",");
                }
                if(n == 4){
                   classify();

                   //return;
                  }
                
              }
  
             }else{
              
              Serial.println("ERR");
             
              }
 
          }else{
           i++;  
          }
          
        }

        a_last = a;
        knock = 0;
        //lock();

            
     if (switchCharacteristic.written()) {
     //Serial.print("got value from app: ");
     //Serial.println(switchCharacteristic.value());
        if (switchCharacteristic.value()==48) {   // any value other than 0
          Serial.println("LED on, value:");
          
          lock();         // will turn the LED on

        } 
        if((switchCharacteristic.value()==49)){                              // a 0 value
          Serial.println(("LED off, value:"));
          
          unlock();          // will turn the LED off
        }
        //batteryLevelChar.writeValue(lockState);
        LockStatusBLE.writeValue(lockState);

        
      }
      
    }

  Serial.println("discon");
  Serial.println(central.address());
  
}
}
/**
 * @brief      Printf function uses vsnprintf and output using Arduino Serial
 *
 * @param[in]  format     Variable argument list
 */

 
void ei_printf(const char *format, ...) {
    static char print_buf[1024] = { 0 };

    va_list args;
    va_start(args, format);
    int r = vsnprintf(print_buf, sizeof(print_buf), format, args);
    va_end(args);

    if (r > 0) {
        Serial.write(print_buf);
    }
}

void classify(){
    // when the central 
    ei_printf("Edge Impulse standalone inferencing (Arduino)\n");
    if (sizeof(features) / sizeof(float) != EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE) {
        ei_printf("The size of your 'features' array is not correct. Expected %lu items, but had %lu\n",
            EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE, sizeof(features) / sizeof(float));
        delay(1000);
        return;
    }
    
    ei_impulse_result_t result = { 0 };

    // the features are stored into flash, and we don't want to load everything into RAM
    signal_t features_signal;
    features_signal.total_length = sizeof(features) / sizeof(features[0]);
    features_signal.get_data = &raw_feature_get_data;

    // invoke the impulse
    EI_IMPULSE_ERROR res = run_classifier(&features_signal, &result, true /* debug */);
    ei_printf("run_classifier returned: %d\n", res);

    if (res != 0) return;
    // print the predictions
    ei_printf("Predictions ");

    float maxVal = 0;
    int match = 5;
   
    //ei_printf("Classification: %d ms",result.timing.classification);
    for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
    ei_printf("%.5f", result.classification[ix].value);
    
    if (result.classification[ix].value > maxVal && result.classification[ix].value > .79){
      maxVal = result.classification[ix].value;
      match = ix;
     }
    }
    
    Serial.println(" ");
    Serial.print("match:");
    Serial.print(match);
    Serial.print(" percent:");
    Serial.println(maxVal);
  
    if(match == 0){
      unlock();
    }
    LockStatusBLE.writeValue(lockState);
    //batteryLevelChar.writeValue(lockState);

  }

void lock(){
    
    lockState = 1;
    digitalWrite(22, LOW);
    digitalWrite(23, LOW);
    digitalWrite(24, LOW);
    digitalWrite(25, LOW);

  }


void unlock(){

    lockState = 0;
    digitalWrite(22, HIGH);
    digitalWrite(23, HIGH);
    digitalWrite(24, HIGH);
    digitalWrite(25, HIGH);

  }

TapLock - A bike lock with machine learning