JetFire

India is an agriculture driven country. With over 280M tonnes of food crops and over 310M tonnes of fruits, the agricultural produce every year exceeds 600M tonnes every year in India. With this much of food being produced, one would expect food to be cheap and every Indian full and statiated. But that is sadly not the case...

According to FAO 2019 estimations, over 194M Indians are malnourished/under fed. This is around 14.5% of the total country population. That's an alarming number considering the fact that its agricultural produce makes it the 4th largest food producer worldwide.

Why? and how?

Well, there are a lot of problems associated with food production in general, and food production specifically in India -

• Food is a highly perishable commodity
• Widespread pest infestations and diseases
• Lack of proper agricultural practices
• Use of low quality seeds, fertilisers, pesticides, etc. reduces crop yields drastically
• Lack of motivation, returns and incentives for farmers and food providers result in suicides, labour shift and low output.
• Outdated technology and continuation of age old traditional practices.
• Unreliable weather conditions and depleted soil makes it difficult to grow crops on the same land without employing crop rotation or other sustainable agriculture techniques.

So what now?

Well, solving such a big problem would be a herculean task. We need to first focus the problem a bit, develop a hypothesis for the solution and then build it. So here is how we narrowed down the problem.

Problem at hand

• No technology use: Currently, farmers spray heavy amounts of pesticides over the field, without taking into consideration any factors or scientific analysis. The quantity and time of spraying is purely based on the judgement and experience of the farmer.

• Important factors neglected: This is very ineffective because the farmer seldomly takes into account many important factors like soil nutrients, weather conditions, crop type, etc. as this is very difficult for the human brain to process and take action on so many factors simultaneously.
• Curing something produces bad side effects and is always less effective than taking precautions: As the pesticides are sprayed after the farmer notices the infestation or when he thinks the infestation will occur, the farmer has to spray pesticides over the entire farm instead of small parts of it where the infestation started. This damages the farm land significantly as the soil nutrients diffuse much faster, and become susceptible to rainwater wash over after getting mixed with the pesticides. This renders the land unusable and barren after a couple of years.

• Resistance Development: Adult pests are difficult to kill, due to their chitinous armour and large numbers. This results in unsatisfactory results and instead increases the resistance of the pests after they survive the pesticide and breed.
• Decreasing green content, increasing toxicity: Resistances in pests makes the farmer increase the toxicity or the quantity of the pesticide he uses. This significantly depletes the green content of the crops, and hence more food is required to provide the same level of nutrition. The toxicity in turn increases, making new diseases and poisoning cases to appear frequently in the population. Here's a foreign article highlighting the same in Germany.
• Good insects are lost as collateral damage: The widespread abuse of pesticides also kills good insects (the ones that help in pollination and seed dispersal) and the bad ones alike, resulting in delay or reduction in pollination and natural forestation.

Wheew!! That was a lot of problems in a cascading effect!! Like a domino, they completely cripple the agricultural produce quality and quantity in a jiffy...

But as they say, the solution lies in the problem itself... Cascading means if the problem is curbed in the nascent stage itself, none of the disastrous outcomes that follow as a result will occur.

Our Solution -

So one solution that can be proposed is to kill the pests before they cause any infestation. This means killing them at egg stage, since their larva cause the majority of damage, eating 2-5 times their own weight daily before growing into the pupa stage. Then why haven't people already done that? Well, cause obviously its not that simple -

• The eggs are small. Very small. Hence they are almost invisible to the naked eye, unless you wish to scan an entire farm of hectares and acres leaf by leaf with a magnifying glass...

• They are not laid out in the open, to be found. They are laid in cracks, crevices, folded leaves, underside the leaf, etc... etc... where they won't be visible and are also hard to reach. Hence normal visible imaging is useless to find them.
• Killing them is very difficult due to their tough chemical resistant shells. Hence either we need to predict when they will hatch (which is not long after they are laid, in case of insects) or we need to spray pesticides that remain effective for long.

So how do we intend to solve this? Well the answer is ..... (drumrolls!!) ..... THERMAL IMAGING!!!

So here's the main idea -

There are a couple of thermal imaging solutions that are in the market. Until recently, this market was completely dominated by FLIR and FLUKE based imaging solutions. However, off late some cheaper low resolution cameras have been released for thermal imaging. These include AMG8833(8x8 pixels) and MLX90640(32x24 pixels) from Panasonic and Melexis resp. This makes it possible for us to develop a solution that is affordable to farmers or the Govt. for large scale deployment.

Using these thermal Cams, we will be taking images of insect egg clusters. This works in our favour because -

• Finding the hidden pest eggs should be possible using thermal imaging: Thermal imaging exposes pest egg clusters irrespective of where they are laid, since plants have low thermal signatures, while eggs have much higher ones due to their high biological activity (cellular division, respiration, etc.). This means that by adjusting the contrast between the temperature spectrum, the egg clusters can be easily classified and identified.

• Eggs are laid as clusters: Since pests mostly lay their eggs in clusters, they become more identifiable even at low resolution and longer distances.
• Good insects are saved: Insects have a tendency of laying eggs on the food of their larva. Hence it can be safely assumed that any egg laid on a crop leaf will be of a pest and useful insects will have a remote colony, away from the farm. This means that our approach will be like a "surgical strike" for the pests, where there will be no/minimal collateral damage to the good insects.
• Increase in green content and mortality: Larva stage is the weakest stage, most vulnerable and easiest to kill. The farmers can easily tell from the egg infestation patterns when they will hatch into larva. Hence, the farmer can directly destroy the infestation from its source, without letting it spread and without using high quantities of toxic chemicals. Even low quantities will be sufficient in producing high mortality rates and prevent them from breeding or developing a resistance.

Dataset Collection

The dataset building proved to be a daunting task. There are no pre collected datasets made available that are suitable for my application. Hence I had to make my own dataset using the AMG8833 camera, which again was imported and hence took weeks to arrive.

I then searched the net to find how to interface the thermal camera with the Jetson Nano. In this process, I found a wonderful library by adafruit that helped me use bicubic interpolation to increase the resolution of the output image. Using this library, I created a raw dataset using a few egg clusters I found below the leaves of a garden plant.

The pre-processing and other tasks took time as well. Once the raw dataset was collected, it was processed(interpolated bilinearly), augmented, annoted, labeled, cropped and resized. This data is then properly segregated and stored in folders to create the preprocessed dataset.

The dataset collection itself proved that no one had tried to do this even on a global search. That gave me even more motivation that I was trying to do something new and novel. Now I had the dataset I needed to proceed further.

Neural Net building

So while the entire world recommended python to me for building the code, being a MATLAB fanatic, I thought it would be worth a try to use MATLAB for this project, because

1. MATLAB has loads of help. For everything. You just need to know what you want to search.

2. MATLAB has innumerable functions, so you have to write minimum code even to perform heavy tasks. As I always say, "If you are clear about what to do, MATLAB will always have a single line function to do it."

3. MATLAB does not have to keep a track of libraries and other dependencies. You just need to download the necessary toolboxes.

4. It becomes easier to perform simulations, training, interfacing HW, make GUIs etc as MATLAB becomes a one stop solution to everything, thanks to SIMULINK, GUIDE, etc. in it.

5. I am well versed with MATLAB, and it has NEVER disappointed me. Neither did it, this time!

So I searched and well, voila! In a very recent 2018b version, they had added a special add on that made it possible to directly generate CuDA code using MATLAB, to deploy a CNN made on MATLAB directly into the root folder of the Jetson Nano, and in return continuously monitor it real time using ROS.! (You can find details here) It sounded amazing and I couldn't wait to try it!! NVIDIA has detailed it here.

To build the Neural Net, I considered using a simple design:

After building the net, I spent quite some time tweaking the parameters to achieve high accuracy. Finally, I achieved almost 100% accuracy, which is expected due to the symmetric and circular shape of the egg clusters.

***Future Scope

Then, using the method demonstrated in this video and the help available online, I developed the optimised CuDA code for my Jetson Nano.

I then used a test a test image to check the classification and it was correctly classified as healthy.

Future Scope

• This project proves to be the proof of concept for a very novel idea and approach, which can change the Indian agriculture significantly. However, this is just a nascent stage in its development.
• I intend to use FLIR and NVIDIA AGX Xavier to train a large number of high quality images on a much more complex net. This will make a pretty robust model.
• Then I will use the MLX90640 and Jetson Nano to deploy it on a UAV, like this one that me and my friend developed.
• I also intend to make a selfie stick like mechanism to help people scan manually, without having to go for costly UAVs.
• This project prove to be challenging due to the lack of resources and datasets on thermal imaging, but I enjoyed it nevertheless and hope to develop this idea even further in the future.