Sober Drive Enforcer

Traffic Jams everywhere!!!

Traffic jams are something very mundane in the city dweller`s life. The American Transportation Research Institute estimates that the total cost of congestion in the freight sector is as high as $74.1 billion annually with $66.1 billion occurring in urban areas. Unsurprisingly, when other indicators such as pollution and accidents are factored in, congestion can cost cities billions of dollars every year. INRIX estimates that in the United States at least, drivers lost more than $88 billion due to tailbacks in 2019 with the average cost for each of them coming to $1, 377. Apart from the associated cost, air and noise pollution is created on a large scale. Often, the reason for a traffic jam is essentially irresponsible parking or improper and unruly driving. There are traffic policemen actively manning city roads for such issues. However, it is not practical to deploy personnel to each and every crossing to get hold of mischief-mongers. To this end, I would like to create a platform to help traffic law enforcement personnel help identify reasons for a traffic jam and get it cleared in the least possible time.

What can be done!!

The solution is a web app that gives realtime analytics about congestion on city roads and gives geotagged information to traffic policemen. Every car is compulsorily connected to the cloud either through a cellular module or a Wifi transceiver connected to a mobile hotspot. The car`s acceleration and gyroscope data would be logged and the data would be fused with the wheel speed sensor and GPS data to identify misbehaving drivers. A sound sensor module would also be installed to capture honking sound levels to identify traffic hotspots to take steps to reduce congestion. Such a solution does not exist on a large scale and even if it does, it is only for applications like geofencing or fleet management. It is really useful as it brings in a wealth of information for enforcement personnel and public policy makers. The things that can be detected are - - Excessive honking, indicative of a jam - Disobedience of traffic rules like overspeeding - Exact location of cause of gridlock

Solution Implementation

The Quickfeather development board acts as the main controller and takes inputs from the mCube MC3635 accelerometer and Infineon IM69D130 PDM digital microphone on board and the externally interfaced gyroscope module MPU-6050 and the GPS module Geekcreit Neo7M module. The microphone data is fed into a model developed using the SensiML studio to identify events of congestion on a particular location on the map. The sounds recognised are those of honking vehicles. The accelerometer data is then fused with the gyro and GPS data using efficient DSP algorithms accelerated on the FPGA core. These points in the multidimensional state space of the geolocation would be fed into a MIMO model that analyses whether the owner is driving improperly and is appropriately fined The microphone also recognises the sound of a revving engine in the vehicle if the driver hasn`t turned it off while in a traffic jam. This helps save on a lot of fuel that is unnecessarily lost.

Things used to make this a reality

Software

• SensiML Studio
• TinyFPGA

Hardware

• Quickfeather development board
• SIM700L module
• Geekcreit Ublox NeoN7M GPS module
• MPU-6050 module connected in SPI mode

Data Collection

Making custom firmware from the repo for audio data recording

In order to use and make the quickfeather have the ability to capture sound we need to make the firmware on our own.

In this tutorial I am using two Operating System which are windows 10 and Ubuntu 18.04 running on WSL. I also install the latest Python 3 software on the windows 10 that is necessary for uploading the firmware

Firstly we need to clone the qorc SDK, (install latest version of git from https://git-scm.com/downloads)

C:\>git clone --recursive https://github.com/QuickLogic-Corp/qorc-sdk

Open your Ubuntu subsystem and change the directory to the qorc SDK

$ cd /
$ cd mnt/c
/mnt/c$cd qorc-sdk
/mnt/c/qorc-sdk$ _

on the qorc-sdk directory do the following Pre-requisite installation steps from the qorc-sdk Repo

Download tarball according to the system configuration from: https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm/downloads

Current stable version tested with is 9-2020-q2-update

• Download tarball according to the system configuration from: https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm/downloadsCurrent stable version tested with is 9-2020-q2-update

Extract the tarball to a preferred path(/BASEPATH/TO/TOOCHAIN/)

sudo tar xvjf gcc-arm-none-eabi-your-version.tar.bz2 -C /BASEPATH/TO/TOOLCHAIN/

The usual preferred path is for example /usr/share

sudo tar xvjf gcc-arm-none-eabi-9-2020-q2-update-x86_64-linux.tar.bz2 -C /usr/share/

• Extract the tarball to a preferred path(/BASEPATH/TO/TOOCHAIN/)sudo tar xvjf gcc-arm-none-eabi-your-version.tar.bz2 -C /BASEPATH/TO/TOOLCHAIN/The usual preferred path is for example /usr/sharesudo tar xvjf gcc-arm-none-eabi-9-2020-q2-update-x86_64-linux.tar.bz2 -C /usr/share/

Add the /BASEPATH/TO/TOOLCHAIN/gcc-arm-none-eabi-your-version/bin/ to PATH (only for current terminal session)

export PATH=/BASEPATH/TO/TOOLCHAIN/gcc-arm-none-eabi-your-version/bin/:$PATH

For the preferred path of /usr/share and current tested stable version 9-2020-q2-update for example:

export PATH=/usr/share/gcc-arm-none-eabi-9-2020-q2-update/bin/:$PATH

• Add the /BASEPATH/TO/TOOCHAIN/gcc-arm-none-eabi-your-version/bin/ to PATH (only for current terminal session)export PATH=/BASEPATH/TO/TOOLCHAIN/gcc-arm-none-eabi-your-version/bin/:$PATH
For the preferred path of /usr/share and current tested stable version 9-2020-q2-update for example:export PATH=/usr/share/gcc-arm-none-eabi-9-2020-q2-update/bin/:$PATH

If the path settings need to be permanent, it can be added to the ~/.bashrc or ~/.bash_profile.

Follow the step completely since it all needed to make our custom firmware

Next still on the ubuntu change the directory to the qf_ssi_ai_app

/mnt/c/qorc-sdk$ cd qf_apps/qf_ssi_ai_app/
/mnt/c/qorc-sdk/qf_apps/qf_ssi_ai_app/$ _

In this folder we need to tweak a few file in order to make the quickfeather audio recording mode active. Here are the modification that you need to do based from Chris Knorowski post. To edit the file you can use any text/code editor you want. For this I am using Notepad++

• Open the file qf_ssi_app/include/sensor_ssss.h
• Modify Lines 75 & 76 to enable SENSOR_SSSS_RECOG_ENABLED

• Open the file sensor_audio/inc/sensor_audio_config_user.h
• Modify Lines 29 & 30 to enable audio SENSOR_AUDIO_RECOG_ENABLED

• Open the file inc/Fw_global_config.h
• Modify the line 78 & 79 to enable the AUDIO_SEnSOR_SELECT_AUDIO

After you modify all the above then we are ready to make the firmware by doing the following step.

Navigate to the GCC_Project folder and then type make

/mnt/c/qorc-sdk/qf_apps/qf_ssi_ai_app/$ cd GCC_Project
/mnt/c/qorc-sdk/qf_apps/qf_ssi_ai_app/GCC_Project$ make

Wait for the firmware to compile, if you getting any error to repeat Pre-Requisite step above. If anything goes accordingly you will have the firmware on the GCC_Project/output folder. Try to type ls on that folder and search for qf_ssi_ai_app.bin

/mnt/c/qorc-sdk/qf_apps/qf_ssi_ai_app/GCC_Project$cd output
/mnt/c/qorc-sdk/qf_apps/qf_ssi_ai_app/GCC_Project/output$ls

If you find the firmware file then congratulation you have make the firmware on your own and Ready for the Next Step.

UploadingThe audio recognition firmware

For this Step I am back on my windows 10 os and using Python 3 on the Command prompt. So on the command prompt Navigate to the qorc-sdk folder and go to TinyFPGA-Programmer-Applicatiion folder

C:\> cd qorc-sdk
C:\qorc-sdk> cd TinyFPGA-Programmer-Application
C:\qorc-sdk\TinyFPGA-Programmer-Application>

This folder have the files necessary to upload the firmware that we make before. Before uploading the firmware we need to make the quickfeather into the upload mode by pressing the RST button then hold the USR button for 3-4 second until the onboard LED turn green then open the Device Manager on the Windows 10 to check serial/COM address of the quickfeather

back to the command prompt do the following command to upload the firmware to the quickfeather board

C:\qorc-sdk\TinyFPGA-Programmer-Application> python tinyfpga-programmer-gui.py --port COMX --m4 C:\qorc-sdk\qf_apps\qf_ssi_ai_app\GCC_Project\output\qf_ssi_app.bin --mode m4

wait around 1-2 minutes for the program to upload the firmware. after all that Finish the we are ready to capture the audio data

Data Capture Lab setup

Open new project and name it

Setting up the quickfeather device sensor to capture audio data

Segmenting and Labelling

• After you finish capturing the data now you need to segment the data
• choose the data
• configure the labelling session, you can choose manual or auto if you are confidence with your data
• choose the segment where you think something is happening at certain time
• After repeating all the process to all data recorded now we are ready to train the data

Model Creation And Testing

Preparing the data

Open the analytic studio and login with your account

https://app.sensiml.cloud/

Building the model

Set the adjustment, this part is trial and error in order to get the best model that you want

• Explore the result of your training
• if you are not satisfied with the result you can try re-train the model and change the setting
• Test the model using test data collected
• Download the Model

Now we are ready to download launch the model back to our device

Deploy it on the Quickfeather DK

after you download the model repeat the step above to upload the firmware back to your quickfeather device