The typical day of every human being starts with waking up late and getting ready for work in a hurry. The workplace will not be very near, one has to travel a significant distance or around 30 minutes on an average. People choose different modes of transport to commute from their home to the workplace and vice versa. Just by driving 30 mins to and fro people get tired like hell due to traffic and bad road conditions especially in developing nations like India.
Now imagine the situation of cab drivers and truck drivers who drive at least 14 to 16 hours a day. In developing nations like India the more attention is given to cost-cutting and high production rate and not to ergonomics and anthropometry of seat design. Due to this reason drivers face fatigue at the end of the day and feel drowsiness while driving. This causes accidents and loss of innocent lives. Last year 1.25M people died due to road accidents. The main cause of these road accidents is nothing but bad road conditions and distracted driver.
Driver distraction may happen due to many reasons such as attending a call physically, try to fiddle with entertainment system, drowsiness due to fatigue etc. By image processing, we can solve the problem of driver distraction by alerting him or her with haptic and voice feedback via speakers. The image processing techniques such as face tracking and drowsiness detection can be implemented using OpenCV or Deep Neural Network on Spresense Board.
FeaturesIGo will have following features:
1. Speed limit alerts
2. Drowsiness detection and alerting
3. OBDII data logging along with GPS coordinates
4. KML map generation
5. Road damage detection using IMU data
And many more coming soon...
Let's see why these features really matters.
Speed limit alerts: This will help the driver to know the speed via voice feedback, thanks to Sony Spresense board. This helps is reducing the driver distraction since driver has to look down to get speedometer reading. This also helps in maintaining or restricting speed in the case of over speeding in urban areas which is also a major cause of accidents. Also it is helpful in rental cars where there will be strict restriction over speed limits.
Here's a part of a report by European Commission for Road Safety
Higher Speeds, More AccidentsHigh speed reduces the possibility to respond in time when necessary. People need time to process information, to decide whether or not to react and, finally to execute a reaction. At high speed the distance covered in this period is longer. At high speeds the distance between starting to brake and a complete stand still is longer as well. The braking distance is proportional to the square of speed (v2). Therefore, the possibility to avoid a collision becomes smaller as speed increases. This is well illustrated at a broad average level by Finch [24].
1 km/hincrease in speed→ 3% increase in accidents
In practice the relationship is more complex. The exact relationship depends among many other things on speed level and road type.
The higher the speed, the steeper the increase in accident risk
The relationship between speed and accident risk is a power function: With increasing speed, the accident risk increases more as the absolute speed is higher.
Based on the principles of kinetic energy and validated by empirical data, Nilsson [44][45] developed the following formula:
In words: the number of injury accidents after the change in speed (A2) equals the number of accidents before the change (A1) multiplied by the new average speed (v2) divided by the former average speed (v1), raised to the square power.
Also road type affects the relationship speed -accident risk
Drowsiness detection: Accidents due to drowsiness and driver fatigue contributes to 20% of overall accidents. These accidents results in severe injury and death in 50% of the times. This happens due to over working driver with lack of sleep. This sleepiness may lead to slow down in information processing resulting in accidents. The drowsiness can also be caused due to alcohol consumption before driving. Since we have camera and voice based feedback system through Sony's Spresense board we can alert the driver using voice feedback. We can train the Deep neural network model with face tracking and drowsiness detection images and get the model to distinguish between alert and drowsy driver.
OBDII data Logging with GPS coordinates: On Board Diagnostics II port helps in tapping all the important data and engine parameters of the vehicle. Analysing this along with GPS data gives us beautiful insights about road conditions, driving behaviors in different regions around the world. Efficient driving i.e changing gears at right RPMs to get most fuel economy etc. All these data are logged in SD card along with time stamp from accurate clocks of GPS satellites. This serves as blackbox in the case of accidents and we can get the data from SD card to see the aftermaths of the accidents. This will also help Insurance companies to catch fake insurance claimers. All the data can be sent to a central server just by adding an ESP 8266 wifi module to this project. This will help in easy and routine diagnostics of the vehicle.
KML map generation : Keyhole Markup Language is a file format used to display geographic data in an Earth browser such as Google Earth. KML uses a tag-based structure with nested elements and attributes and is based on the XML standard. All tags are case-sensitive and must appear exactly as they are listed in the KML Reference.
Road damage detection using IMU data: Roads are not very well maintained in developing nations like India, due to which there will be number of potholes on the road which are nothing but death traps in dark and busy morning times. People try to avoid potholes and lose balance and die due to accidents. Sony Spresense board along with IMU MPU6050 can read the accelerometer values and can get the details of pothole locations based on dip in the value read by the sensor. This helps in geo-tagging the pothole and there by alerting the authority to fix it ASAP. This can be very easy and revolutionary way of collecting the data.
Step 1: Get the PartsPlease get the parts mentioned above. PAM 8403 amplifier for speakers is not required if you already have a portable speaker 3.5mm Jack. Or you can even plug it to the AUX input of a vehicle's entertainment sytsem
Step 2: Let's Connect Them1. Connect the HC-05 Bluetooth module as shown in the figure. It will get connected to the ELM 327 module by pairing. This can be achieved by sending AT commands to bluetooth module and configuring it to connect directly to ELM327 OBDII dongle.
2. Connect the IMU Sensor MPU6050 to I2C pins i.e SCL and SDA pins of Spresense extension board and power it with 3.3V. We will use 3.3v throughout our project so we can switch the jumpers to the 3.3V side in the expansion board. It will be good for bluetooth module also as it communicates to controller at 3.3V logic levels.
3. Connect the Speakers to the audio port i.e 3.5mm jack, male sure that the speakers are powered externally.
4. Connect the microphone along with biasing resistor to MIC A of the extension board as shown in figure. This MIC will be used to record in the case of emergency message or as a blackbox message. I had an old servo in not working condition I just removed that and soldered it to mic.
5. Put the formatted SD card and install all audio encoders decoders and.mp3 files for speed limit warnings. This can be generated from online text to voice converters like this. Text to.mp3
Step 3: Let's Test EverythingConnect the board to the PC and upload the code written in Arduino IDE this code does following things, Arduino Code is self explanatory as I have written all the required comments.
1. Reads data from OBDII port wirelessly over bluetooth by querying different AT commands
We can bind our ELM327 OBDII dongle to HC05 so that we can have a seamless connection. For that we will have to enter the AT command mode by holding the button on the Bluetooth module and connecting the power supply.
You can use FTDI Bridge to talk to HC-05 module.
Get your ELM 327's Bluetooth address by connecting to your phone. My ELM's address is 00:1D:A5:68:98:8C
AT+RESET
AT+ORGL (Set to original)
AT+ROLE=1 (Set to Master)
AT+CMODE=0 (Set connect to a specific address)
AT+BIND=1D,A5,6898C
AT+INIT (Need to connect)
AT+PAIR= 1D,A5,6898C,20 (20 means 20 second timeout)
AT+LINK=1D,A5,6898C
but while sending you will have to ignore zeros i.e 1D,A5,6898C
2. Logs all the values on the SD card.
3. Logs the values of road damage level along with the time-stamp and location i.e Latitude and Longitude from on board GNSS receiver in.CSV format.
4. Alerts driver as he crosses different speed levels through voice feedback
Now let's go to the image processing part of the project. Since I stay in India I received the Spresense boards 4 days ago i.e 27th Feb 2019. I started working on it as soon as I got it. After several efforts of debugging and asking questions in forum I had no luck with the camera. I tried to train the model on the Cloud environment but since the camera was not working It will be of no use. Mostly camera might got damaged during transport itself, Because I had taken careful measures while opening in order to protect it from static discharges. Forum thread
So I thought of doing the same with a laptop using python and OpenCV. I was finally able to make it work properly and its here. This will apply facial landmark localisation to extract the eye part from the face. We will have to set a certain threshold value below which the aspect ratio is considered as a blink. Here EAR stands for Eye Aspect Ratio.
def eye_aspect_ratio(eye):
# compute the euclidean distances between the two sets of
# vertical eye landmarks (x, y)-coordinates
A = dist.euclidean(eye[1], eye[5])
B = dist.euclidean(eye[2], eye[4])
# compute the euclidean distance between the horizontal
# eye landmark (x, y)-coordinates
C = dist.euclidean(eye[0], eye[3])
# compute the eye aspect ratio
ear = (A + B) / (2.0 * C)
# return the eye aspect ratio
return ear
You may need to install openCV2, Numpy, dlib and Scipy packages to run this.
To run the program you need type following command,
$ python .\drowsiness_detection.py -p .\shape_predictor_68_face_landmarks
.dat -a .\alarm.wav -w 1
shape_predictor file is a data set and alarm.wav is the sound whose original creator is Matt Koenig. -w 0 corresponds to builtin webcam of PC and -w 1 corresponds to external webcam.
Step 4: Working VideoRoad damage data : I connected ESP 8266 based Node MCU board to plot the data. Here's a result of the same. These logged data sets can be monetized for machine learning applications as data is more valuable in this data driven world.
1. Training the DNNRT model for drowsiness detection and implementing it on Sony Spresense board
2. Adding an LCD display to the project.
3. Adding antenna bu doing board level modifications
References:
Comments