Interactive Driving Interface and Online Tracking

Today, even as the technology developed, motor accidents are still on the rise. A major fraction of accidents is resulting due to the carelessness of the driver. This is majorly noticed when the concentration is lost from the road, even for a split of a second. Whenever a motor accident has happened, getting immediate medical attention at the earliest can decide the difference between life and death.

With our smart solution, relevant information on traffic, routes and vehicle data are provided to the driver using an attached pair of headphones. It also incorporates a variety of additional functions such as traffic and road analysis, object and environment detection using the accompanying Rpi and USB camera, hands-free communication with gadgets, audio output, etc. This helps the driver to concentrate on his driving rather than changing his attention to other things. The advanced features such as object and environment detection on the fly will also improve his situational awareness. Additionally, this solution can be easily deployed to any kind of motor vehicles.

Spresense is a compact development board based on Sony’s power-efficient multi-core micro-controller CXD5602. It allows developers to create IoT applications in a very short time and is supported by the Arduino IDE as well as the more advanced NuttX based SDK.

The advantage of using this board for the embedded projects is the simplicity of programming such a powerful platform using a readily available platform such as Arduino and the subsequent compatibility. In this project, we will exploit the GNSS and audio capabilities of the Spresense board family.

Keywords: Driving Assistant, Sony Spresense, Raspberry Pi. Embedded System

-----------------------------------------------------------------------------------------------------------------

Project Goals

• Gather Driving information
• Driver Alerts
• Update tracking information
• Advanced lane detection and analysis
• Object and environment detection on the fly
• Hands-free communication and user interaction

-----------------------------------------------------------------------------------------------------------------

Workflow and Explanation

This project guide explains all the basic steps and procedures that are performed towards attaining the final goals of the project. It wont be expressedin highly technical contexts,but hadtried to keep this as simple as possible so that anyone referring to this guide would be able to infer and understand the concept. This guide ismade as compact as possible for the sake of readability. Wherever applicable, appropriatereferences will be made to resources in other websites where detailed information can be obtained instead of repeating everything here. However, the most important points will stillbe mentioned.

The project is divided into three parts:

1. Setting up Sony Spresense System

2. Setting up Raspberry Pi System

3. Assembling the prototype

For tinkering with the Sony Spresense Unit, there are some pre-requisite steps that needs to be carried out. Click the respective links for a detailed walk-though to perform each step.

• Setup the Arduino IDE for Sony Spresense
• Setup Audio for Sony Spresense

After setting up the pre-requisites, upload the idiot.ino sketch into the Spresense board. The code basically polls the GPS system for updated position in the form of current latitude and longitude as provided by the GPS/GLONASS systems and give necessary information and alert to the driver/user.

What basically the idiot.ino program does is that while driving, it constantly checks for the GNSS information. When the GPS coordinates matches the pre-defined destination coordinates, it alerts the driver the same via an audio message. The serial communication takes place at a baud rate of 115200.

//include gnss functionality
#include <GNSS.h>
  
//initialize GNSS class
static SpGnss Gnss;
  
//starts the GNSS
result = Gnss.begin();
  
//Selects GPS satellite system
Gnss.select(GPS);
  
//starts GPS positioning
result = Gnss.start(COLD_START);

Tip: For getting a good GPS fix, recommend to test the device outdoors in an open area.

After setting up the SD card for the audio service, insert it into the micro SD card slot of the Spresense Expansion Board.

//include Audion functionality
#include <Audio.h>
  
//initialze Audio class
AudioClass *theAudio;
  
//start Audio system
theAudio = AudioClass::getInstance();
theAudio->begin(audio_attention_cb);
  
//define player settings
theAudio->setRenderingClockMode(AS_CLKMODE_NORMAL);
theAudio->setPlayerMode(AS_SETPLAYER_OUTPUTDEVICE_SPHP, AS_SP_DRV_MODE_LINEOUT);
  
//start player
theAudio->initPlayer(AudioClass::Player0, AS_CODECTYPE_MP3, "/mnt/sd0/BIN", AS_SAMPLINGRATE_AUTO, AS_CHANNEL_STEREO);
theAudio->writeFrames(AudioClass::Player0, myFile);
  
//stop player
theAudio->stopPlayer(AudioClass::Player0);

Tip: In case if you receive an error that says "DSP file ___ cannot open", it is because of some error in setting up the memory card and the files. Please refer here for details.

For the Raspberry Pi, lite versions like the Raspbian OS is preferred. Python programs are developed for obtaining the image processing requirements with the help of the attached USB camera.

Advanced lane detection has been implemented using OpenCV. This works based on the concept of optical flow. Optical flow is the pattern of apparent motion of image objects between two consecutive frames caused by the movement of object or camera. It is 2D vector field where each vector is a displacement vector showing the movement of points from first frame to second.

The advanced lane detection involves the following steps:

For the Prototype development, the Sony Spresense system, Raspberry Pi and USB webcam were mounted on a riding helmet. A standard power bank was used to provide power supply to all the systems,

-----------------------------------------------------------------------------------------------------------------

Testing

Apart from this, the audio warning system has also been tested which provided the audio information via the attached headset upon reaching the destination.

Object detection was also tested along with the Lane analysis with the help of TensorFlow. Unfortunately, Raspberry Pi was not good enough to run the TensorFlow models. So the same was tested on the laptop also. At a later stage, we plan to implement the same into the IDIOT system again after some optimisation.

Link

-----------------------------------------------------------------------------------------------------------------

Planned Future Modifications

• Inclusion of a mobile application to connect, manage and track progress
• Inclusion of a non-obstructive augmented display on the helmet visor to provide visual information regarding the ride along with the already implemented audio information system
• Inclusion of offline voice processing and recognition system to take user input voice for carrying out specific tasks such as finding alternate routes, pull up weather information, traffic density ahead etc.
• Inclusion of object and environment detection into the system.
• In the event of an emergency, provision to automatically or manually alert pre-configured contacts.

-----------------------------------------------------------------------------------------------------------------

Conclusion

Getting hands on the Sony Spresense Camera module was found to be a tough call which made us to satisfy with a standard USB web camera. Adding to that, the delay in shipment and customs issues ended up eating a lot of valuable project time and resources. The journey that we took to realise the project goals and to complete the tasks to achieve them were very challenging. This journey also gave us lot of good experiences. Hoping that this project will motivate and create sparks in some of those little minds to proceed forward, make their hands dirty and innovate.