Trukdoc - AIoT solution for in-transit vehicle monitoring

Contents

· Introduction

o The problem

· Solution

o Summary

o Features

o User Interface

· Implementation

o Technology behind features implemented

o Model Development on Edge Impulse for drowsiness detection

o Model Development on Edge Impulse for drowsiness detection

o Deployment of Zephyr RTOS on Nordic Thingy : 91

o Cloud Architecture

§ Accessing data on Monogoto IoT Console

§ Forwarding data to webhook.site

§ Using IFTTT for alerts

· Testing

o Deployment of node on actual truck

o Visualisation of GPS coordinates

o Alert Mechanism

· Future Improvements

The Problem

Fleet owners in India suffer a huge loss of time and money because of unwanted breakdowns and sudden component failures in their trucks. This is caused by the undetected problems in truck components due to the ignorance of both the fleet owners and truck drivers, the high cost of regular maintenance and undesirable downtime experienced with each checkup. Breakdowns consequently lead to major downtime, exacerbated by a lack of mechanical assistance on highways and at night. Another common issue faced by Indian truckers is drowsiness due to long journeys at night hauling cargo, which leads to accidents and deaths.

Despite being such a critical component of the Indian supply chain, the trucking industry is plagued with problems ranging from monetary and safety constraints to environmental issues. Today, many fleet management solutions exist which solve the problems encountered by a fleet owner in pre-transit and post-transit. However, there is a multitude of problems faced by the driver and the owner in the most critical part of the whole journey i.e. the “in-transit” part, which remain unsolved.

Poor truck maintenance and driver drowsiness are the major causes of truck accidents. Trucks were involved in 13% of all road accidents in India in 2018, which amounts to around 57000 accidents, and 150 thousand casualties.

A video interview with a truck driver was what opened my eyes to this huge problem.

The Solution

With the above problem definition, it is clear that there is a pressing need for an in-transit vehicle telematics system specialised for Indian trucks. We studied several competitors in the vehicle telematics sector and found that none of them is solving the on-ground problems faced by Indian fleet owners and truck drivers. They are either creating generic solutions catering to all vehicles which cannot solve specific problems present only in Indian trucks or providing telematics for newer models of trucks which don’t cater to the 7.6 million trucks running on Indian roads today.

Through our suite of solutions, we bridge the gaps in the existing solutions to reduce costs incurred per truck and increase the safety of drivers in transit. Our solutions include driver drowsiness detection, DEF (Diesel Exhaust Fluid) level detection and coolant pipe monitoring, tyre pressure and wheel alignment monitoring, brake and clutch health monitoring and axle bending detection, with an overarching feature of updates provided even without legacy cellular connectivity through the NBIoT protocol.

Additionally, I benchmarked our solution with international players such as ScanGauge and FieldLogix where problems specific to the Indian market such as lack of connectivity and urea detection were not addressed. My findings have been presented in the following table.

From the detailed analysis and competitor benchmarking, it is very clear that there is a gap in the Indian trucking industry and consequently a need for the comprehensive yet user-friendly IoT-based solution that we are offering.

TrukDoc aims to support fleet owners and empower truck drivers by tracking vehicle health in transit to provide early detection of common vehicle problems and driver monitoring, using our truck telematics solution to reduce costs incurred per truck and boost safety through the decreased risk of accidents.

Features

Our solution comes with a wide range of features including monitoring of truck components like the brake, clutch, axle, and tires, detecting driver drowsiness and getting updates even without a cellular network, among others. The following figure lists the key functionalities offered.

The User Interface

A key component of the solution is the user interface that acts as an abstraction layer between the underlying deep tech and the technology-agnostic driver or fleet owner. The three components of the UI are –

• - Dashboard display
• - Website
• - Mobile Application for driver

Driver Application

Driver app informs the truck driver about any component damage and provides nearby mechanic contacts. It is equipped with an assistive UI that provides audio-visual assistance to help drivers in navigating the app. The picture below gives an idea of the look and feel of the app.

A demonstration is shown in the video below.

Dashboard display

In case of component damage or other problems, a warning is displayed on this dashboard installed near the steering, and a red LED starts to blink. The picture below gives an idea of the look and feel of the dashboard (Sorry about my horrible CAD skills).

Website Dashboard for fleet owner

In order to track the location, issues and warnings for all trucks in the fleet, the fleet owner is equipped with a dashboard. Through this, they receive live updates of the location of the truck, component damage issues or warnings and contacts of nearby mechanics in case of vehicle breakdown. The picture below gives an idea of the look and feel of the webapp that can be accessed at -

https://trukdoc.netlify.app/dashboard

The video below shows a demonstration of fleet management and realtime alerts on the website.

Overall Architecture

The overall architecture of the solution is shown in the figure below. It displays a Master-slave architecture where various slave nodes collect sensory data and send it to the master wirelessly which then uploads warnings and key data points to the cloud from where notifications are sent to end-users.

Master Node

The Nordic Thingy:91 was chosen as the processing unit for the MASTER node. The selection criteria in favour of the NRF9160-based microcontroller were its NB-IoT communication features, low cost, rich peripheral set, low sleep mode current, and compatibility with the nRF5 SDK(with which I am quite familiar by now) helping me achieve optimal performance while reducing BOM and time to market.

Software Architecture

The flowchart below explains the algorithm for the master node. All the slaves send warnings to the MASTER if a fault is detected. Some slaves keep sending data to the MASTER at regular intervals based on which secondary checks are performed. The GPS data is continuously streamed. A component check is performed at the beginning of each trip. All the vehicle’s logged health data is analyzed and sent to the cloud through the gateway at the end of the trip.

Software API Description

Slave Nodes

There are five slave nodes in total that monitor different parameters related to vehicle health. The following table gives a summary of the functionalities of the master and the slave nodes.

Let me discuss the features packed into Truckdoc individually and how each of the nodes is realised in practice.

Power Consumption

The power consumption for a particular module is calculated based on its estimated uptime. The uptime is further divided into active & passive phases. In the active mode, most system resources are enabled & wireless transmission links are active. Passively, the microcontroller is sent to the deep sleep mode and woken up at certain predefined intervals of time to send or receive data. The duty ratio quantifies the fraction of the time for which the microcontroller is active. From the current consumption in both the modes, the power is calculated, which is then used to predict the approximate battery life of each sensor node.

The life of almost all the modules is more than one year. After that, the modules can easily be charged through a standard Micro-USB port with a portable power bank. Note that the power consumption of the Master node has not been considered since they are powered directly off the 12V power supply of the truck and only set in active mode on battery power in emergencies when the fleet owner wants to know the truck status while the truck is powered down.

Features

Since the number of nodes were so many, I thought it would be better to take the feature implementations one by one, rather than describing the nodes. The list of features along with the technology is shown below.

Driver Drowsiness Detection

Problem:

Truck drivers in India are often subject to inadequate sleep or have jobs requiring them to alternate sleep cycles which is often taxing. This results in drivers becoming drowsy while driving and potentially meeting with an accident.

Technology:

The dashboard consists of a Raspberry Pi with an infrared camera to monitor the driver's eyes. IR LEDs ensure uninterrupted monitoring even during low-light conditions and at night when most accidents occur. A pre-trained face landmark detection model is used to determine if the eyes are closed or open using EAR (Eye Aspect Ratio). If the eyes remain closed (EAR < 2.0) beyond a specific time interval (200 μs), an alarm is raised to alert the driver. If the number of warnings exceeds a certain threshold, the fleet owner is also alerted.

For training the model, Edge Impulse was used. The entire process is documented in a later section. A neural network based image classifier predicted if the driver`s eyes were open or closed in real-time and sent it over I2C to the Nordic Thingy:91 – the master controller.

Brake System Monitoring

Problem:

For medium and heavy-load transport vehicles, it is critical for the braking system to function correctly and be maintained regularly.

Technology:

A 6 DoF Inertial Measurement Unit (IMU) is interfaced with the MASTER to measure the vehicle’s deceleration when the brakes are pressed. At full braking pedal position, a limit switch interfaced with the MASTER detects a pedal press, after which the acceleration value is polled.

If the deceleration is below the required threshold a certain number of times, an alert is raised with the fleet owner. The threshold value is dynamically adjusted depending on the road conditions by monitoring the acceleration value of the truck over an overlapping time window.

The fleet owner can observe timely monitoring of deceleration magnitudes to decide on maintenance-related issues.

Axle Bending Monitoring

Problem:

Trucks are often loaded beyond the maximum limit, which leads to over-the-mark stresses on the leaf springs and, consequently, the axles of the rear wheel pairs. Prolonged driving with bent axles leads to bending and breaking, leading to fatal accidents.

Technology:

SLAVE nodes fixed at the ends of the axle quantify the bending by measuring the linear displacement of the ends using a Linear Variable Differential Transformer (LVDT) sensor to sense the bending in the axle. A warning signal is sent to the MASTER node if the linear displacement exceeds a certain threshold. The MASTER node then raises an alert with the fleet owner about the vehicle’s overloading.

Clutch Health Monitoring

Problem:

Clutch plate wear due to improper shifting techniques leads to accidents. Late detection leads to higher repair costs.

Technology:

A SLAVE node reads the rotational speed of the input shaft to the gearbox. The position of a magnetic sticker on the gearbox input shaft is encoded using a hall effect sensor (AH49E). The rotational speed is calculated locally and transmitted wirelessly to the MASTER. The MASTER compares the input shaft RPM with the engine RPM received from the ECU OBD bus. If the RPMs differ beyond a certain magnitude for a standard response time interval (1 - 1.5 s), an alert is raised with the fleet owner.

Clutch bites are recorded and stored to assess the health of the friction material over time and suggest minor repairs. The frequency of bites also gives a reasonable measure of the shifting skills of the driver.

Coolant Pipe Monitoring

Problem:

Air bubbles trapped in the cooling system lead to overheating of the vehicle. Due to this, trucks can go over the safe operating temperature range leading to a blown head gasket, cracked engine block, damaged pistons, bursting hoses, or a blown radiator.

Technology:

The SLAVE node has a fluid pressure sensor that relays coolant pressure data to the main module wirelessly via nRF communication at regular intervals. This data can be used to detect possible leakage and further analyze the condition of the coolant fluids. The driver and the fleet owner are intimated if immediate servicing has to be done.

Tire Pressure Monitoring

Problem:

Operating within an optimum tire pressure range is integral to having good maneuvering characteristics and fuel economy.

Technology:

Our system provides one air pressure sensing SLAVE node (fixed near the nozzle on the rim) per wheel, relaying pressure data to the main module wirelessly via nRF communication at regular intervals. The main module reads the pressure data from all the tires to detect leaks or punctures and informs the driver by a message on the dashboard display about refilling air or getting the tires checked for holes. The fleet owner is also notified about possible flat tire situations to take immediate action.

Wheel Alignment Monitoring

Problem:

Proper wheel alignment is crucial for the control and stability of the vehicle, and misalignment can lead to severe accidents..

Technology:

Improper alignment is detected by observing the variation of the rotation of the front tires about the ends of the steering rack. Our module uses three rotary encoders - one for the steering wheel rod and one each for the two front wheels - to measure rotation angles. The angles turned by the wheels and the angle turned by the steering rod are individually calculated by the respective SLAVE nodes.

and sent wirelessly to the MASTER node. The MASTER node correlates the steering rotation in a particular direction to the rotation of the individual front tires in that direction. Discrepancies beyond tolerable limits are reported to the fleet owner on high priority with a suggestion for a wheel alignment inspection.

Fuel Quality Monitoring

Problem:

An optimum air-to-fuel ratio is required for the engine’s clean and efficient combustion of fuel. The engine efficiency is heavily degraded by suboptimal grade fuel which is often substituted by opportunistic drivers in place of recommended fuel.

Technology:

The lambda sensor (connected to the ECU) measures the amount of unburnt oxygen in the exhaust gases. The MASTER module taps into the ECU OBD bus to read this data and analyzes it over time to calculate the fuel quality and engine efficiency. Failure to reduce levels of unburnt oxygen by the ECU (by air-to-fuel ratio calibration) indicates fault either in ECU or fuel pump. This is notified to the fleet owner, and an alert for maintenance is raised.

DEF Level Monitoring

Problem: Heavy and medium load vehicles generally use diesel for fuel. Combustion of diesel leads to a significantly high release of NOX in the exhaust gas, which is highly hazardous. Hence, exhaust gas is treated with DEF (Diesel Exhaust Fluid) to reduce NOX emissions. DEF consists of urea and deionized water. The ammonia released by the urea solution reacts with NOX to produce much less harmful gases (N2, CO2, & H2O) into the exhaust. If the level of DEF decreases below a certain threshold, the concentration of harmful gases in the exhaust increases polluting the air and attracting hefty fines from pollution control enforcement.

Technology:

The ammonia sensor (MQ-135) in our SLAVE node detects urea depletion in the DEF tank. The ultrasound sensor (HC-SR04) monitors the liquid level and assists in assessing the evaporation of the liquid from the DEF tank. Combined inputs from both the sensors are analysed on a timely basis and the fleet owner is alerted regarding the DEF replacement as and when required.

Model Development using Edge Impulse

Edge Impulse was used to develop two different neural networks. The first one was for detection of driver drowsiness while the second one was detection of harsh driving by the driver. For the first model, imagery data was collected manually using the data acquisition tab in Edge Impulse. For proper training of the model, I collected images of ten different people (my friends, of course) with eyes closed and open. To make it much more realisitic, I collected images with heads bowed down, sideways and even wearing sunglasses!!

After the data was collected, I created a standard train/test split of 79% is to 21% and moved on to training. Something important to note here is that the data has to be labelled during collection. Otherwise, it becomes a pain to view each sample individually on the console and tag it.

For the training of the data, I first designed the impulse. I used the recommended settings as it had brought good results to many hacksters in the past. I scaled the images captured to 96x96 size for reducing the feature size. After that I used a Transfer Learning block which is essentially a transfer learning network that generates two outputs – open or closed indicating whether the driver`s eyes are open or closed.

For the next step, I tweaked some parameters to train the model. Here comes the interesting part. I had to juggle through a lot of parameters to finally arrive at the best implementation. I must say that the EON Tuner helped a lot. I used the MobilNet V2 with a 0.1 dropout as the final model since it gave the best tradeoff of inference time, which is the time taken by the model to process results and RAM usage. Note that as the dropout is increased, the size of the network and its accuracy increases. As you would see from the EON Tuner results, increasing the depth of the networks does not always results in better accuracy.

Remember that the Raspberry Pi just has 512MB of RAM out of which a significant portion is needed to run the operating system as well. From my experiments, I advise keeping the RAM usage below 200 KB. The Model size or the Flash usage is not much of a concern for the Raspberry Pi, but it was a concern for the Nordic Thingy used for the Driver Behaviour Prediction Model.

As you can see from the picture below, I achieved an accuracy of over 90% which is great for a minion of a processing platform, that is the Raspberry Pi Zero.

Finally, I used the EON Tuner, which is a proprietary tool inside EI to optimise the final model for the target hardware. I chose a Linux Development board with similar specifications as the raspberry Pi zero and ran the tuner. I must warn you that it took the tuner six hours to find the best model but its was worth the wait. As the screenshot below shows, the EON Tuner goes one step ahead in throwing visual warnings whenever the model developed is unsuitable for the target hardware.

After you are satisfied that the model performs optimally, quantise it to reduce its size and then test it using the test dataset. As you can see, my testing results also give an accuracy of 90% which is great for this application. The inference time is also just 3 ms which makes the model practically realisable on the target hardware.

The most exciting part was the final deployment. I deployed the model as a Linux binary which generated a model file that could be used directly with the Edge Impulse C++ SDK for Linux.

I did not realise that I was the first person to be actually deploying an image processing application using EI. Hence I did not have much guidance. Thankfully, the documentation was good and I was able to find my way through. The crux of the deployment was vectorising the image feed from the Raspberry Pi camera and feeding it to the ‘features’ vector for the model to start inferencing. As you would see in the final testing, the model works in the real world too.

For the second model, I took the acceleration feed from the Nordic Thingy over 10 minutes and tagged samples based on different driving behaviour. To keep it simple, I just kept two output features – good and harsh driving. For harsh driving, the criteria were sudden changes in acceleration which occurs on harsh braking or overacceleration. I trained and deployed the model using a similar approach as described earlier. The only change is that I deployed the final model as a Zephyr library. I did this since I was using Zephyr RTOS on the Nordic Thingy:91. There was not particularly a reason to do this but I felt that since the Thingy was already crunching a lot of data from a lot of sources it made sense to have a dedicated OS.

Putting it all together

Setting up alert mechanism using IFTTT

The first step is to create a free account on IFTTT at https://ifttt.com/

Click on Create to create the applet. The following screen appears

Click on Add and search for Webhooks in the search bar. Click Webhooks. The following screen should appear.

Click on Receive a web request with a JSON payload. This is done to receive the JSON file created using AT commands. The file is forwarded from the webhook URL provided to IFTTT and finally used to configure the alert message.

Give a name to the trigger and click on Create Trigger. On the next screen enter the phone number where the message has to be sent.

Make sure to write a suitable message to convey meaningful information to the end user. In my case, I have tried to capture harsh driving/overspeeding events and alert the fleet owner.

There is just one final thing that needs to be done. Open up webhook.site in a browser and get the following screen.

Copy the unique URL and paste it in the AT command sent out to send HTTP messages to the cloud.

Setting up the Nordic Thingy : 91

Follow along the tutorial : https://www.hackster.io/psu-capstone-pj3c/cellular-iot-irrigation-system-for-onem2m-nordic-thingy-91-5cfc14

To set up Zephyr RTOS on the Nordic Thingy : 91.

Once done, modify the example GPS tracker application shipped with the Nordic Thingy to communicate with the webhook set up using AT commands. There is enough documentation available online to understand how to send messages. In brief, the following sequence of messages needs to be sent.

Remember to call the Edge Impulse inference library deployed for driver behaviour estimation and that’s literally it for the code!!! Onto testing now.

Results and Testing and “Road”-testing….

The product was put to test in a real-world scenario by installing it in a ramshackled truck on a metalled in India (a random garbage truck with a very welcoming driver). The five slave nodes were installed at their respective locations and sensory data streamed to the master node was observed.

Slave 1 was fitted inside the lid of the DEF tank

Slave 2 was fitted at the ends of the front axle

Placement of slave 4

Placement of the LVDT sensor interface

Slave 4 was fitted near the engine compartment and was by far the trickiest of the lot. As I mentioned, I am not great at mechanical design and designed a pretty poorly designed compartment. Of course, there was a time constraint as well. This stupid design also took the 3D printer a whooping 12 hours(I use a Creality Ender 3 Pro).

Slave 3 was fitted on the ends of the casing of the rear axle. The fitting was really tricky as I had to improvise even after making the casing to scale.

Slave 5 was fitted on the nozzle of a rear tyre

One of the slave node`s internal structure. The rest of the slaves are nearly the same. Its just the sensor that changes. In the image shown, the battery, the Arduino Uno, the NRF24L01+ node and the sensor, which in this case is a rotary encoder.

The final prototypes for the MASTER node tested on an actual moving truck are shown below.

Master Node – sitting high and handsome on the truck dashboard

Lets hit the Road

After a bit of tuning, the engine was fired and we went for a ride!!!

All the while, the GPS location of the truck was streamed at intervals of one minute. It was driven on a straight road for around a minute and faults were simulated artificially. For the sake of demonstration, two of the features were tested as shown in the testing video (tested the others too but it is kind of similar; so I removed it for sake of removing redundancy).

First, the driver was asked to act drowsy in front of the steering wheel. It was observed that the drowsiness detection mechanism was able to detect the condition really well and was able to raise an alarm in time.

For the next test, the tyre was deflated slowly(Did this for the first time and it was a huge tyre, almost my size). As soon as the pressure dropped below the set threshold, a warning was displayed on the dashboard.

The following video shows the entire testing procedure as well as the placement of nodes

SMS message alerts are sent out from the IFTTT platform as soon as an anomaly is detected. Note that the anomalies detected here were created artificially as demonstrated in the video.

Future Work and Improvements

I must say I am pretty happy with how things have panned out given the tight deadline while managing the storms in my academic and professional life. The prototype works just fine and has been roadtested on an actual truck, which validates the usefulness of the product. However, it is still far from commercial deployment. I believe the Nordic Thingy is a great piece of hardware for such IoT applications and I thoroughly enjoyed working on it. As good as the implementation may be, there is always scope for improvement. Some of the improvements I would like to add are –

- A full-fledged mobile application with more advanced security features. Right now, I am using Monogoto IoT Cloud Platform with AES-256 bit encryption which is a bit of an overkill for this application. I would like to explore more efficient implementations with a different cloud platform like Ubidots.

- Additional slave units to capture more sensory data so that more problems can be solved. On the wishlist are lubrication monitoring and drunken state detection from a breath analyser.

- More robustly trained neural networks to draw more accurate inferences and prevent false alarms. For the sake of saving time, models obtained using the Edge Impulse tool were used with minimal changes to the settings. This would be changed in later iterations by tweaking the internal parameters of the network(of course with help from my “AI-expert” friends).

- Reduction of latency in the alert mechanism framework so that end users can take quick action in case of a mishap

- Replacing Arduino Uno used in the slave nodes with a more professional low-power alternative like the EFM8BB microcontroller from Silicon Labs(refer to the attached schematics). Though the Arduino Uno performs well functionally, it draws a significant amount of current which is not desirable, given the small battery pack powering the node. Further, the cost of the final solution can be reduced significantly by using this barebones microcontroller.

- Replacing the NRF24L01+ transceiver on the master node by using the onboard NRF52832 microcontroller on the Nordic Thingy that can also communicate using the legacy nRF Shockburst protocol. As mentioned earlier, I did give it a try but was not able to make it work in time.

- Improved mechanical design. I am clearly not very good at it and I might need to take help from some of my “CAD-expert” buddies. Right now, the modules are not IP67 protected, which is needed in the rough environments in which it is going to be deployed.

- Writing ISO-26262 compliant code so that I can get the product licenced for commercial use. (Anyone from Nordic reading this is requested to guide me as to whom to approach regarding this. I am already aware of your ecosystem. Maybe you can connect me with some of your partners)

Conclusion

It was a thoroughly enjoyable exercise working on the Nordic Thingy:91 for this design challenge. I thank Hackster.io for providing me with this platform and helping me develop a tangible product which may well be the foundation of my dream startup. I know I have a long way to go with this project. Any advice on the existing design or the path forward is welcome and appreciated. Thanks a lot for reading this looooooooooooooooooooong article!! Hope I was able to keep you hooked.

Thanks,

Moinak