Predictive Maintenance in Industry 4.0 with Sony Spresense

Motivation behind this project

It all started when I saw an industrial accident video where a person got winded up in an industrial packing machine; for about 15-20 minutes, no one around could help him, and there was no accident indication. This incident happened a few years ago, and I did not have any technical expertise to make a solution, but I thought if there was some auto-detect mechanism or at least a notification method integrated, that person could have been saved from the accident.

Fast forward to 2021, when IIT Madras Pravartak launched a grand challenge competition in collaboration with Sony India and DST, Govt. of India, in which I registered with the problem statement "Predictive maintenance". The journey started in July 2021 when the competition page link was shared in the Makerville group.

We worked on this project for the next eight months starting from abstract submission to final submission and grand final ceremony on 22nd March 2022.

And we won! Started from 700+ registrations, abstract idea selection, quarter-final selection, semi-final selection, final selection, and final demonstration in the grand ceremony in front of IIT Madras professors, Sony India, and Japan engineers.

Project Details

Problem statement stated by Pravartak for the predictive maintenance is as follows:

Predictive maintenance is a way to minimize both failures and maintenance work, while maximizing machinery and component lifetime. Predictive maintenance relies on “condition-monitoring” to provide data about the current performance of the machine, in order to predict issues and prevent failures. A solution that performs predictive maintenance in High Temperature Process Infrastructure in Industries like Petro-chemical, Manufacturing, Energy etc. could be very useful.

To make a suitable project for this problem statement, the core fundamentals of industrial machines and maintenance were essential to integrate with the latest technology like EdgeML and real-time predictive maintenance.

This project is divided into two parts:

1. Product Design

The core concept behind this project is to develop a product that can be easily integrated into industrial machines and applications without voiding the guarantee or any other physical aspect of the machines.

Considering the modular approach of Industry 4.0 and the nature of our initial prototype, we developed a 3D printed enclosure to house Sony Spresense main board, extension board, camera module, microphone module, and other sensors.

Hardware Architecture:

Third Party Integrations:

3D design of enclosure:

Some essential hardware integration included microphone module, light indicator, aluminum extrusion mount and lights.

Light Indicator is a small DIY andon light with WS2812 LEDs:

Closest industrial machine on which prototype was tested is my 3D printer:

2. Integration

Product design is a comprehensive solution of the integration developed using the Edge Impulse platform and custom integration.

Integration/software is further discussed:

• Custom python integration:
  Edge Impulse provided compiled binary to flash on the sony spresense directly, and there was no option to grab data and use it for custom integration quickly; hence we made a python script that picked up data from sony spresense serial port and analyzed data for further integration.

• Implementation:
  Our solution statement for the competition was:"AI-Enabled real-time decision making & predictive analysis. Maximizing productivity, ensuring safety, and analyzing real-time data to predict asset's life intelligently"To achieve this in the prototype, we started the process of tinyML implementation, which was based on the software block diagram:

• Classifications
  1. Image Classification
  This classification differentiates normal conditions from any other unwanted interaction with the machine. This can be extended to industrial applications with unwanted human interaction or machine failure due to unwanted hindrances.

ML model is validated with testing new data and transfer learning.

Available on Edge Impulse Studio - https://studio.edgeimpulse.com/studio/75310

2. Object Detection

This classification is used to classify unwanted output from machines; in the prototype classification of spaghetti, this can be extended to industrial applications where the result is not according to the master condition.

Edge impulse released FOMO [Faster objects, More objects] which provide awesome results [Approx 477ms / 1-2 FPS]. Earlier object detection was used and results were not good.

Available on Edge Impulse Studio - https://studio.edgeimpulse.com/studio/75846

3. Noise Detection

When there is an anomaly in machines, the sound is the primary classification which can be any unwanted sound and further result in noise. In this prototype, faulty fans are being detected, and in terms of practicality, two faulty fans were identified while making data sets!

This classification can be extended to industrial applications dealing with noise detection and finding faulty parts before a big breakdown occurs!

Classification was based on three RAW data filtering as follows:

Available on Edge Impulse Studio - https://studio.edgeimpulse.com/studio/63123

4. Sensor Data Acquisition

Additional sensor data is collected to improve data points and provide additional details of failure or features like geotagging.

In this prototype BME280 is attached which provides Temperature, pressure and altitude data.

5. Dashboard

A simple UI must showcase real-time data to showcase the final data. In this prototype, the dashboard show data acquisition in real-time and human-readable data.

Temporarily hosted on: https://anshumanfauzdar.me/predictive-ui/

Video Demonstration

Detailed Documentation:

Everything documented in detailed manner using notion and hosted on : https://sony.anshumanfauzdar.me

Final Verdict

Apart from 3D printer, prototype was tested on industrial machines and validated by experts.

The journey of making this project has been quite exciting and helped me learn a lot of new things, and now we are looking forward to launching this project into a product as soon as possible.

Let's connect:

➔ Twitter

➔ LinkedIn

➔ GitHub