CliSensio - Climate Sensing and Insect Infestation Control

Overview:

Climate Change, a highly debated high school topic, but do we really care about it? We see politicians, scientists, researchers, philanthropists, conservationists and almost everyone blaming each other for every climate catastrophe occurring today. The real question is what climate change or change are we talking about?

Change is never easy for any species or life form but it happens in order to survive and evolve and this is the change we need in order to challenge climate change - the change could be in form of laws to reduce carbon emissions, conserve biodiversity, advanced technologies which can outperform existing techs in terms of greener energy or total innovative revamping of traditional ways of all human activities.

According to soil scientists, at current rates of soil destruction (i.e. decarbonization, erosion, desertification, chemical pollution), within 50 years we will not only suffer serious damage to public health due to a qualitatively degraded food supply characterized by diminished nutrition and loss of important trace minerals, but we will literally no longer have enough arable topsoil to feed ourselves. Without protecting and regenerating the soil on our 4 billion acres of cultivated farmland, 8 billion acres of pastureland, and 10 billion acres of forest land, it will be impossible to feed the world, keep global warming below 2 degrees Celsius, or halt the loss of biodiversity.

Important Note: This project was the result of a discussion with my parents on environment conservation especially saving food crops, beneficial insects and supporting our farmers. My father made me understand a very critical thing- "We all know the changes happening, we all can see climate catastrophes and loss of living organisms and their habitats but we haven't made much effort in saving those because over the past decades we have only made detection and warning systems when the world needs a self activation systems to control the situation." Those words were like North Star for my project development. Also, this project was initiated on May 20, World Bee Day - aiming to protect the unsung heroes of nature. My innovation and solution here is to 'adapt and mitigate' and not to 'detect and forget'.

What is our CliSensio?

CliSensio - Climate Sensing and Insect Infestation Control is a new adaptive device which runs SensiML AutoML audio recognition on the edge to recognize disease spreading vectors, hungry pests and threatful species. CliSensio device firmware can be flashed for mosquito species identification, pest, rodents identifications and elimination responses.

Background:

1. Infestations of Insects turning Farmland into Deserts:

A peer-reviewed study which stated that the threat of rising temperatures and unpredictable weather patterns has multiplied tremendously. By 2050, droughts and insect infestations could destroy 26.2 percent of the World's largest tea exporter, Kenya's tea growing areas. A research study concluded that climate change is more dangerous than we can imagine, it could empty 'World's Grain Bowls' - Rising temperatures will stimulate insects' appetites -- and make some prone to reproducing more quickly -- spelling danger for key staples like wheat, corn and rice which feed billions of people.

While the world is still reeling from the coronavirus pandemic, several countries Brazil, Argentina, India, Pakistan and other East African nations are facing locust attacks which threaten to trigger food crisis in them.

"When it gets warmer, pest metabolism increases, " said Scott Merrill, a researcher at the University of Vermont and co-author of the study in the journal Science. And when pest metabolism increases, insect pests eat more food, which is not good for crops. Overall, "global yield losses of these grains are projected to increase by 10 to 25 percent per degree of global mean surface warming, " said the report. - http://13.235.153.162/world/global-warming-will-make-insects-hungrier-eating-up-key-crops-study-162721

• France stands to lose about 9.4 percent of its maize to pests in a world that is 2 degrees C warmer, compared to about 6.6 percent of yield losses today due to pests.
• In Europe, currently, the most productive wheat producing region in the world, annual pest-induced yield losses could reach 16 million tons.
• Eleven European countries are predicted to see 75 percent or higher losses in wheat from pests, compared to current pest damage.
• In the United States, the world's largest maize producer, insect-induced maize losses could rise 40 percent under current climate warming trajectories, meaning 20 fewer tons of maize per year.
• China, home to one-third of the world's rice production, could see losses of 27 million tons annually.

Pandemic, war, climate change fuel fears of global hunger, food scarcity. The first challenge is to accept the problem. The second one is to accept responsibility. Climate change is unlike any challenge humanity has faced because unlike pandemics and wars, here we have a warning. We have time to prepare for the impending disaster, but we are not exactly making good use of it. We're arguing and grandstanding, instead of acting. When the oceans rise, there is no fix. When the air becomes unbreathable, masks will not work. - The world needs to come together and find solutions to the climate disaster problems:http://13.235.153.162/world/climate-change-the-action-is-missing-despite-the-high-profile-summits-379354

2.Losing Grassland Birds is another Widespread Ecological Crisis:

Growing unnecessary and competitive use of pesticides and insecticides that sweep out all insects affect the entire food chain of birds. Moreover, the insects are becoming resistant to the chemicals day by day but the birds which eat those insects barely survive for reproduction. Using lots of chemicals to boost farm yields but we tend to forget that due to excessive use of those pesticides, insecticides and inorganic manures, cross pollinating agents, the arthropods( 90% of living organisms found on earth are insects) don't get attracted to those plants or otherwise die due to excessive chemical exposure and pollution. More than 2, 000 pollinating insects are extinct by now and only 500 are left to become extinct if not protected - "Even if there is no insect disease in farms, farmers uselessly spray chemicals and kill useful insects like honeybees and other birds too."

More than 90 per cent of the losses are from 12 species including sparrows, warblers, blackbirds, and finches.

"Now, we see fields of corn and other crops right up to the horizon, everything is sanitized and mechanized, there's no room left for birds, fauna and nature." - Ken Rosenberg, an ornithologist at Cornell University and principal co-author of the paper in Science told AFP. The National Observatory of Biodiversity estimates a 30 percent decline in grassland birds between 1989 and 2017. The number of birds in the United States and Canada has fallen by an astonishing 29 per cent, or almost three billion, since 1970, scientists reported: https://www.wionews.com/world/north-american-bird-population-fell-by-quarter-over-50-years-study-250699

3.Eliminating threat species to our pollinators(our best friend, bees):

Insects are pollinators and waste management experts, controlling other insects, keeping the soil healthy while also serving the most important function – being food for those higher up the food chain. In the absence of pollinators, many plant species will disappear.

Flowers can hear buzzing bees—and it makes their nectar sweeter

Sure, the argument can be made that humans will be able to recreate such plants in the laboratory. Approximately 87 per cent of the over 300, 000 species of plants need external pollinators, where would one begin? Pollinators are our secret weapon in the fight against global warming.

Also, scientists are now on a hunt for the hornets, hoping to eradicate the species before they wipe out US bees. According to the scientists of the Washington State University (WSU), "it's a health hazard, and more importantly, the shockingly large hornets are a significant predator of honeybees." They will begin trapping queen murder hornets this spring, aiming to detect and eradicate the species.

Populations of honeybees and other pollinators in the US were already under pressure.

The United States has been fighting a deadly virus. It is now the world's worst affected country and now it is preparing to fight another enemy, except this one is not invisible. On the contrary, it is huge. The world's largest hornets have been spotted for the first time in America.

4. Billions at risk of mosquito-borne diseases:

Climate Change Will Expose Half of World’s Population to Disease-Spreading Mosquitoes By 2050. According to a study from PLOS Neglected Tropical Diseases by the end of this century almost all of the world's population would be exposed to mosquito-borne diseases. Aedes aegypti and Aedes albopictus are two species of mosquitoes known to carry and transmit potentially deadly viruses like dengue,chikungunya,Zika and yellow fever. These bugs can thrive in urban environments, often traveling alongside humans as we ship goods and travel around the world.

And, according to the study, these tropical mosquitoes are poised to expand their ranges, exposing a billion additional people to the diseases they carry in the next 50 years. It’s important to note that this study is on a global scale, and mosquito management efforts will need nation-by-nation or even city-by-city information while trying to combat the viruses. Detecting mosquito breeding grounds is more difficult which is one of the reason for such widespread population of mosquitoes.

With a changing climate and extreme weather like floods expected to become more frequent and intense, this could mean more outbreaks of mosquito-borne disease such as dengue in the coming years. New mosquito species would pose more burdens on public health system. https://news.mongabay.com/2021/03/as-climate-change-brings-more-floods-mosquito-numbers-could-swell-study/

Current Solutions VS CliSensio Solutions:

1. As far I have researched the only potential solution in long term for insect infestation control is to develop a model to predict breeding areas that may have been missed by ground monitoring- ''The focus will be on stopping hoppers becoming adults, as that leads to another cycle of infestation. We want to avoid that. We want to advise governments early before an upsurge happens." So far it has attracted an investment of £35 million by the United Kingdom as part of its Weather and Climate Information Services for Africa program($60m but, if an upsurge occurs, the cost will soar to $500m) to forecast the movement of locusts using data such as wind speed and direction, temperature, and humidity.

Resource-deprived countries have already started deploying aircraft and drones to spray insecticide on the swarms to deal with the situation. But figuring out the exact time and location of the swarms has remained a challenge for them.We need something small, feasible and as first order of defense to prevent future locust or other insect outbreaks: - Our CliSensio device is here to solve this challenge..

2-3. For saving the grassland birds and beneficial organisms, the only solution is to avoid unnecessary use of insecticides and pesticides. Our CliSensio device will stop the competition among the farmers for using excess fertilizers, pesticides, insecticides to boost production.Using the correct farm audio analysis the device would auto-assist and spray insects/pests repellents with precise localization in the farm without chemical overdosage or wastages and thus save farmers money, efforts and stop degradation of environment.

4.Most mosquito-breeding spots are found in homes and not at construction sites. This makes the job of detecting breeding spots harder because, unlike entering construction sites, the authorities cannot go into homes easily or frequently to check on them. We need such solution which works on ground level autonomously for everywhere including home. Our device CliSensio has a special feature to combat this problem at community and household levels. Source:https://www.straitstimes.com/singapore/why-detecting-mosquito-breeding-spots-is-difficult

Let's Build it!

Hardware Setup:

The heart of the device is QuickFeather EOS3 https://www.quicklogic.com/products/soc/eos-s3-microcontroller/ All SensiML AI algorithms are going to be performed on this device. See all below images, for device setup.

How is liquid-chemical atomizer utilized in Clisensio?

The liquid atomizer has two components, a piezoelectric atomising head and a driver circuit. A wick is used for supplying the chemical liquid to a piezoelectric atomising head arranged in the spray apparatus. In CliSensio, the devices is used for chemical liquid spraying for targeted species,i.e. insect removal, insect growth retardation, insect repellent, disinfection based upon SensiML audio recognition and feature extraction models trained to detect harmful pests and insects.

Furthermore, the present innovation relates to a method for controlling harmful organisms by effectively and economically releasing active ingredients having the functions of insect removal, insect growth retardation and insect repelling in the air.Stopping harmful insect and pest breeding in their growth period, saving farmers and communities from any perils.The method would massively reduce overdosage of chemicals used in traditional agriculture or insect control.

Device Firmware Setup:

The device firmware is based on QORC-SDK, https://github.com/QuickLogic-Corp/qorc-sdk all instructions are well documented here.

Clone this QORC SDK repository using the shown command, and check if make and GCC compiler is set correctly or not (if not follow the steps in github link README.md from above link).

$ git clone --recursive https://github.com/QuickLogic-Corp/qorc-sdk
# run gcc -v and make -v to find gcc compiler toolchain and make utility
# versions on your system
$ gcc -v & make -v

You can use any code editor, Eclipse C/C++ IDE is preferred but other IDEs can be used too(I used VS Code as it is easier and swiss knife for my projects).

All the data extraction and training has been done using SensiML Analytics Studio and SensiML data capture lab: https://sensiml.com/products/ Currently data capture lab is only for windows platform, that's one downside for linux and Mac users, but I managed it since I have both systems at my hand(windows and linux separately not WSL). Our CliSensio device can run two different firmware:

1. Mosquito Species Detection and Control 2. Pest Detection and Control

1. Mosquito Species Detection and Control

• Data Collection: Since we are using microphone as the only data source for our CliSensio (I had planned to bring in audio as well as sensor data into the project but currently either sensors or audio ML models are supported at a time, not both together.) The audio dataset were already collected, this project is inspiration from the other projects I made in past to solve pest and insects problems, links here: 1. https://www.hackster.io/vilaksh01/tensorcrop-crop-quality-and-farm-control-3831b8 2. https://www.hackster.io/vilaksh01/sigfy-sigfox-farm-yield-with-crop-and-health-monitoring-3d8116All audio data used is uploaded on this github page: https://github.com/vilaksh01/CliSens.io/tree/main/CliSensioMosquito-audio_data However, I recommend you to record it via your own device through SensiML data capture lab to achieve more accuracy, as those audio files are not recorded on same microphone architecture and bit rate.
• Data Segmentation: This was performed through Data capture lab after data collection step. I recommend to use Audacity to check spectrogram for the audio data before proceeding to segmentation, it gives clear analysis about noise-to-valuable data in audio file.

• Training and Saving model: Log into your SensiML cloud or Data Analytics Studio to check our SensiML data and proceed training steps. We trained on two pipelines to make model more efficient and accurate. Pipeline 1 Check below images carefully for all steps.

Pipelines for our model: I tried testing two pipelines for mosquito breed detection with different Optimization Metric and Feature Extractors. Pipeline 1, is default parameters with f1-score as optimization metric whereas Pipeline 2 has Sensitivity as Optimization Metric and unknown classifier parameter is allowed with custom feature generator set.

Advantage of using Sensitivity as Optimization Metric (Ratio of true positives to total or actual positives in the data.) Important when: Identifying the positives is crucial.Used when: The occurrence of false negatives is unacceptable/intolerable. You’d rather have some extra false positives (false alarms) over saving some false negatives. For example, when predicting financial default or a deadly disease.

• Visualizing Model Summary: This is one of the most important and highly rewarding feature for ML training engineers(they get to know their models better). Below are the summary for pipeline 1 and pipeline 2.

• Download the model: This is the final step to download Knowledge pack for Mosquito Breed Classification and Control for our firmware.

Our generated model json file for Pipeline 2, check the class maps- 1. Aedes Aegypti- is known to transmit dengue virus, yellow fever virus, chikungunya virus, and Zika virus. It is suggested to be a potential vector of Venezuelan Equine Encephalitis virus, and is capable of transmitting West Nile virus., 2. AedesAlbopictus- This mosquito species is a known vector of chikungunya virus, dengue virus and dirofilariasis., 3. AnophelesMinimus- association with malaria, filariasis and arbovirus infections., 4. Background- Noises like wind, white noise, etc., 5. Culex Tarsalis- spread the viruses that cause West Nile fever, St. Louis encephalitis, and Japanese encephalitis, as well as viral diseases of birds and horses., 0. Unknown Classifier;

{
"NumModels": 1,
"ModelIndexes": {
"0": "Pipeline_2_rank_4"
},
"ModelDescriptions": [
{
"Name": "Pipeline_2_rank_4",
"ClassMaps": {
"1": "AedesAegypti",
"2": "AedesAlbopictus",
"3": "AnophelesMinimus",
"4": "Background",
"5": "CulexTarsalis",
"0": "Unknown"
},
"ModelType": "PME",
"FeatureNames": [
"channel_0PctTimeOverZero",
"channel_0PctTimeOverThreshold",
"channel_0Mean",
"channel_0NegativeZeroCrossings",
"channel_075Percentile",
"channel_0Sum",
"channel_0ZeroCrossingRate",
"AvgSigMag",
"channel_0TotArea",
"channel_0TotAreaDc"
],
"AIF": [
93,
33,
23,
131,
52,
54,
47,
86,
51,
76,
120,
61,
113,
83,
85,
80,
64,
87,
23,
52,
36,
56,
75,
25,
24,
71,
107,
23,
20,
22,
23,
137,
76,
43,
20,
28,
53,
44,
23,
133,
24,
22,
48,
120,
95,
117,
26,
21,
41,
27,
123,
25,
30,
112,
132,
67,
46,
20,
46,
20,
52,
61,
78,
58,
30,
25,
107,
45,
78,
134,
160,
160,
160,
160,
160
],
"Category": [
4,
2,
3,
4,
5,
4,
5,
2,
5,
3,
4,
3,
5,
5,
5,
3,
3,
4,
2,
3,
5,
2,
3,
2,
2,
1,
2,
1,
1,
1,
2,
2,
5,
2,
2,
1,
3,
5,
2,
3,
1,
1,
2,
4,
3,
4,
1,
2,
2,
2,
2,
2,
1,
3,
2,
5,
1,
2,
3,
1,
2,
4,
2,
1,
2,
1,
5,
2,
1,
1,
2,
2,
1,
1,
3
],
"Vector": [
[
208,
208,
210,
52,
89,
210,
58,
210,
210,
15
],
[
59,
59,
180,
90,
19,
180,
84,
181,
180,
21
],
[
196,
196,
195,
57,
35,
195,
65,
195,
195,
3
],
[
30,
30,
196,
0,
22,
196,
47,
196,
196,
1
],
[
152,
152,
197,
24,
35,
197,
22,
197,
197,
0
],
[
169,
169,
195,
2,
24,
195,
32,
196,
195,
2
],
[
195,
195,
197,
20,
27,
197,
41,
197,
197,
0
],
[
123,
123,
181,
62,
26,
181,
79,
181,
181,
21
],
[
197,
197,
197,
61,
29,
197,
91,
197,
197,
1
],
[
154,
154,
194,
13,
25,
194,
71,
195,
194,
3
],
[
100,
100,
195,
0,
23,
195,
31,
195,
195,
3
],
[
198,
198,
194,
110,
51,
194,
110,
194,
194,
3
],
[
132,
132,
197,
16,
69,
197,
14,
197,
197,
1
],
[
201,
201,
197,
104,
33,
197,
126,
197,
197,
0
],
[
212,
212,
197,
25,
85,
197,
25,
197,
197,
2
],
[
175,
175,
195,
79,
34,
195,
89,
195,
195,
3
],
[
204,
204,
193,
34,
118,
193,
33,
193,
193,
6
],
[
152,
152,
195,
0,
24,
195,
16,
195,
195,
3
],
[
65,
65,
181,
59,
20,
181,
57,
181,
181,
21
],
[
182,
182,
195,
38,
29,
195,
40,
195,
195,
3
],
[
199,
199,
197,
39,
29,
197,
59,
197,
197,
0
],
[
121,
121,
181,
178,
25,
181,
180,
181,
181,
21
],
[
147,
147,
194,
45,
30,
194,
37,
194,
194,
4
],
[
50,
50,
180,
74,
19,
180,
70,
180,
180,
22
],
[
49,
49,
181,
59,
19,
181,
67,
181,
181,
21
],
[
110,
110,
180,
134,
24,
180,
128,
180,
180,
21
],
[
201,
201,
181,
41,
45,
181,
47,
181,
181,
20
],
[
61,
61,
180,
45,
19,
180,
56,
180,
180,
22
],
[
36,
36,
180,
49,
18,
180,
46,
181,
180,
21
],
[
52,
52,
181,
53,
19,
181,
55,
181,
181,
21
],
[
30,
30,
180,
54,
18,
180,
45,
180,
180,
22
],
[
164,
164,
180,
87,
40,
180,
86,
180,
180,
21
],
[
229,
229,
194,
16,
60,
194,
16,
195,
194,
3
],
[
114,
114,
181,
37,
25,
181,
33,
181,
181,
20
],
[
78,
78,
180,
78,
21,
180,
67,
180,
180,
22
],
[
64,
64,
180,
52,
20,
180,
45,
180,
180,
22
],
[
214,
214,
195,
67,
40,
195,
66,
195,
195,
3
],
[
194,
194,
197,
54,
37,
197,
62,
197,
197,
0
],
[
74,
74,
180,
48,
20,
180,
41,
180,
180,
22
],
[
182,
182,
194,
121,
44,
194,
123,
195,
194,
3
],
[
95,
95,
181,
37,
22,
181,
50,
181,
181,
21
],
[
78,
78,
181,
52,
21,
181,
56,
181,
181,
21
],
[
94,
94,
180,
39,
22,
180,
35,
180,
180,
22
],
[
208,
208,
202,
36,
111,
202,
44,
202,
202,
6
],
[
193,
193,
194,
67,
42,
194,
70,
194,
194,
4
],
[
189,
189,
204,
50,
67,
204,
56,
204,
204,
10
],
[
80,
80,
181,
52,
20,
181,
40,
181,
181,
20
],
[
51,
51,
180,
66,
19,
180,
52,
180,
180,
22
],
[
59,
59,
181,
24,
18,
181,
26,
181,
181,
21
],
[
44,
44,
180,
32,
19,
180,
25,
180,
180,
22
],
[
148,
148,
181,
39,
45,
181,
38,
181,
181,
21
],
[
15,
15,
180,
7,
17,
180,
7,
181,
180,
21
],
[
80,
80,
180,
131,
21,
180,
124,
180,
180,
22
],
[
230,
230,
194,
52,
54,
194,
52,
194,
194,
4
],
[
83,
83,
180,
141,
21,
180,
137,
180,
180,
21
],
[
183,
183,
197,
24,
37,
197,
22,
197,
197,
1
],
[
78,
78,
181,
76,
21,
181,
80,
181,
181,
21
],
[
14,
14,
180,
21,
17,
180,
17,
180,
180,
21
],
[
196,
196,
193,
33,
55,
193,
32,
193,
193,
4
],
[
53,
53,
181,
48,
19,
181,
40,
181,
181,
21
],
[
59,
59,
180,
62,
19,
180,
53,
180,
180,
22
],
[
193,
193,
196,
4,
25,
196,
9,
196,
196,
1
],
[
80,
80,
180,
65,
20,
180,
47,
180,
180,
22
],
[
37,
37,
181,
64,
18,
181,
60,
181,
181,
21
],
[
35,
35,
180,
37,
18,
180,
28,
180,
180,
22
],
[
31,
31,
181,
23,
18,
181,
25,
181,
181,
21
],
[
197,
197,
195,
17,
55,
195,
15,
196,
195,
2
],
[
30,
30,
181,
41,
18,
181,
39,
181,
181,
21
],
[
23,
23,
180,
29,
17,
180,
26,
180,
180,
22
],
[
53,
53,
180,
40,
19,
180,
33,
180,
180,
22
],
[
6,
6,
181,
10,
17,
181,
6,
181,
181,
21
],
[
105,
105,
181,
53,
24,
181,
40,
181,
181,
21
],
[
68,
68,
181,
88,
20,
181,
93,
181,
181,
21
],
[
124,
124,
180,
201,
26,
180,
202,
180,
180,
21
],
[
168,
168,
195,
34,
43,
195,
32,
195,
195,
4
]
],
"Identifiers": [
0,
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26,
27,
28,
29,
30,
31,
32,
33,
34,
35,
36,
37,
38,
39,
40,
41,
42,
43,
44,
45,
46,
47,
48,
49,
50,
51,
52,
53,
54,
55,
56,
57,
58,
59,
60,
61,
62,
63,
64,
65,
66,
67,
68,
69,
70,
71,
72,
73,
74
],
"DistanceMode": 0
}
]
}

• Integrating Knowledge Pack with QORC-SDK and our firmware: The app structure is similar to qf-apps/qf_ssi_ai_app in QORC-SDK. First step is to set up our GPIO pin for liquid-atomizer module in src/pincfg_table.c file.

{ // setup Water Atomizer Digital Output
.ucPin = PAD_23,
.ucFunc = PAD23_FUNC_SEL_GPIO_7,
.ucCtrl = PAD_CTRL_SRC_A0,
.ucMode = PAD_MODE_OUTPUT_EN,
.ucPull = PAD_NOPULL,
.ucDrv = PAD_DRV_STRENGTH_4MA,
.ucSpeed = PAD_SLEW_RATE_SLOW,
.ucSmtTrg = PAD_SMT_TRIG_DIS,
},

After setting up GPIO let's modify src/sml_output.c function to direct the action response based upon audio classification or inference performed. Also, from here we can see the serial communication prints out classifications which can be used to send to dashboard for Spatio-Temporal Vector Ecology map over LoRa or cellular, helping governmentto be prepared for unforeseen events(epidemic or pandemic).

static void sml_output_serial(uint16_t model, uint16_t classification)
{
int count;
int wbytes = 0;
int buflen = sizeof(sensor_ssss_ai_result_buf)-1;
int ret;
kb_get_feature_vector(model, sensor_ssss_ai_fv_arr, &sensor_ssss_ai_fv_len);
// you can send other sensors or GPS data in below line for dashboard
count = snprintf(sensor_ssss_ai_result_buf, buflen,
"{\"ModelNumber\":%d,\"Classification\":%d", (int)model, (int)classification);
wbytes += count;
buflen -= count;
switch((int)classification)
{
// case if unknown audio found
case 0:
HAL_GPIO_Write(GPIO_4, 1); //blue LED for unknown
HAL_GPIO_Write(GPIO_5, 0); //green LED off
HAL_GPIO_Write(GPIO_6, 0); //red LED off
break;

// case if mosquito species 'Aedes' 'Anopheles' 'Culex' detected
case (1 || 2 || 3 || 5):
HAL_GPIO_Write(GPIO_7, 1); // chemical atomizer GPIO 7 on
HAL_DelayUSec(15*1000*1000);  // wait 15 seconds for atomizer
HAL_GPIO_Write(GPIO_7, 0); // chemical atomizer GPIO 7 off
HAL_DelayUSec(15*1000*1000);  // wait 15 before resuming
break;

case 4:
HAL_GPIO_Write(GPIO_5, 1); //green LED for background
HAL_GPIO_Write(GPIO_4, 0); //blue LED off
HAL_GPIO_Write(GPIO_6, 0); //red LED off
break;
}

After everything is in place, let's compile and flash the firmware to the device using the below command in project directory.

$ make clean
$ make
$ qfprog --port /dev/ttyACM0 --m4app ./qf_ssi_ai_clisensio_app.bin --reset --mode m4

2. Pest Detection and Control

• Data Collection: We are going to collect audio dataset for different pests, currently for this POC, I am more focused on locust sounds here (as India’s agriculture ministry is hoping to control the invasion before monsoon season hits north India at the end of June, when locusts mature and breed. If the infestation is not controlled, it may threaten summer crops such as rice, maize and sorghum.) However, gathering locust swarm audio data was difficult so I tested with cricket sound files available for free here(Locusts belong to the same order of insects as grasshoppers, katydids and crickets - the Orthoptera). Here, are all the audio data for cricket sound https://mixkit.co/free-sound-effects/crickets/

• Data Segmentation: We have segmented the audio data for two categories, viz. Cricket and Background.

• Training and Saving model: In SensiML Analytics Studio, we are going to create model and train the data for classification. Our query name is: Q_PESTS_SPECIES. Training for Pipeline_1 is shown in one of the below images. Here, I tested two pipelines as before, the other pipeline is also shown below.

• Integrating Knowledge Pack with QORC-SDK and our firmware: Here's our Pest detection and control json file generated:

{
	"NumModels": 1,
	"ModelIndexes": {
		"0": "Pipeline_1_rank_3"
	},
	"ModelDescriptions": [
		{
			"Name": "Pipeline_1_rank_3",
			"ClassMaps": {
				"1": "background",
				"2": "cricket",
				"0": "Unknown"
			},
			"ModelType": "PME",
			"FeatureNames": [
				"channel_025Percentile",
				"channel_0ZeroCrossingRate",
				"channel_0Sum",
				"channel_0100Percentile",
				"channel_0SigmaCrossingRate",
				"channel_0Skew",
				"channel_02ndSigmaCrossingRate",
				"channel_0MeanCrossingRate",
				"channel_0IQR",
				"channel_075Percentile",
				"channel_0Kurtosis",
				"channel_0Mean"
			],
			"AIF": [
				150,
				133,
				150,
				128,
				109,
				150,
				133,
				126,
				105,
				144,
				150,
				116,
				104,
				150,
				121,
				85,
				87,
				93,
				84,
				89,
				86,
				140,
				140,
				100,
				90,
				150,
				89,
				90,
				150
			],
			"Category": [
				2,
				2,
				1,
				2,
				2,
				1,
				1,
				2,
				2,
				1,
				1,
				1,
				2,
				1,
				1,
				1,
				1,
				1,
				1,
				1,
				1,
				2,
				1,
				2,
				1,
				2,
				1,
				2,
				2
			],
			"Vector": [
				[
					231,
					221,
					138,
					22,
					209,
					180,
					107,
					220,
					25,
					25,
					22,
					138
				],
				[
					224,
					79,
					131,
					22,
					95,
					179,
					97,
					79,
					31,
					30,
					18,
					131
				],
				[
					202,
					15,
					227,
					50,
					20,
					169,
					31,
					17,
					59,
					64,
					28,
					227
				],
				[
					250,
					127,
					138,
					6,
					146,
					180,
					168,
					126,
					5,
					5,
					27,
					138
				],
				[
					253,
					43,
					137,
					8,
					66,
					206,
					56,
					45,
					2,
					2,
					91,
					137
				],
				[
					153,
					13,
					115,
					29,
					41,
					180,
					0,
					13,
					102,
					100,
					1,
					115
				],
				[
					212,
					11,
					130,
					36,
					24,
					157,
					10,
					11,
					45,
					47,
					23,
					130
				],
				[
					251,
					55,
					133,
					23,
					59,
					182,
					178,
					56,
					4,
					3,
					59,
					133
				],
				[
					233,
					17,
					129,
					32,
					40,
					181,
					31,
					17,
					21,
					19,
					69,
					129
				],
				[
					248,
					13,
					107,
					28,
					6,
					195,
					15,
					13,
					6,
					5,
					98,
					107
				],
				[
					239,
					3,
					168,
					37,
					7,
					181,
					5,
					3,
					20,
					24,
					84,
					168
				],
				[
					236,
					15,
					118,
					17,
					22,
					148,
					36,
					14,
					19,
					20,
					40,
					118
				],
				[
					253,
					104,
					137,
					2,
					148,
					176,
					110,
					106,
					2,
					2,
					21,
					137
				],
				[
					219,
					15,
					136,
					115,
					7,
					175,
					25,
					16,
					36,
					35,
					127,
					136
				],
				[
					238,
					10,
					124,
					30,
					9,
					188,
					20,
					9,
					18,
					19,
					48,
					124
				],
				[
					217,
					15,
					143,
					14,
					37,
					176,
					10,
					16,
					37,
					35,
					4,
					143
				],
				[
					233,
					12,
					151,
					20,
					19,
					198,
					46,
					12,
					22,
					21,
					21,
					151
				],
				[
					254,
					17,
					146,
					1,
					35,
					170,
					25,
					17,
					1,
					2,
					13,
					146
				],
				[
					238,
					7,
					162,
					8,
					17,
					174,
					10,
					8,
					17,
					16,
					10,
					162
				],
				[
					232,
					9,
					147,
					11,
					24,
					189,
					13,
					10,
					23,
					24,
					8,
					147
				],
				[
					241,
					12,
					135,
					13,
					27,
					190,
					36,
					13,
					14,
					14,
					20,
					135
				],
				[
					232,
					23,
					136,
					15,
					39,
					182,
					31,
					24,
					23,
					23,
					12,
					136
				],
				[
					251,
					38,
					134,
					3,
					63,
					183,
					36,
					39,
					3,
					3,
					17,
					134
				],
				[
					249,
					67,
					138,
					12,
					88,
					179,
					158,
					67,
					6,
					6,
					36,
					138
				],
				[
					251,
					69,
					136,
					7,
					63,
					183,
					107,
					69,
					4,
					4,
					35,
					136
				],
				[
					253,
					67,
					140,
					15,
					45,
					183,
					138,
					67,
					2,
					2,
					178,
					140
				],
				[
					247,
					65,
					143,
					7,
					88,
					176,
					64,
					66,
					9,
					9,
					19,
					143
				],
				[
					252,
					85,
					137,
					2,
					111,
					180,
					92,
					86,
					3,
					3,
					18,
					137
				],
				[
					253,
					69,
					138,
					2,
					99,
					193,
					117,
					67,
					2,
					2,
					18,
					138
				]
			],
			"Identifiers": [
				0,
				1,
				2,
				3,
				4,
				5,
				6,
				7,
				8,
				9,
				10,
				11,
				12,
				13,
				14,
				15,
				16,
				17,
				18,
				19,
				20,
				21,
				22,
				23,
				24,
				25,
				26,
				27,
				28
			],
			"DistanceMode": 0
		}
	]
}{
	"NumModels": 1,
	"ModelIndexes": {
		"0": "Pipeline_1_rank_3"
	},
	"ModelDescriptions": [
		{
			"Name": "Pipeline_1_rank_3",
			"ClassMaps": {
				"1": "background",
				"2": "cricket",
				"0": "Unknown"
			},
			"ModelType": "PME",
			"FeatureNames": [
				"channel_025Percentile",
				"channel_0ZeroCrossingRate",
				"channel_0Sum",
				"channel_0100Percentile",
				"channel_0SigmaCrossingRate",
				"channel_0Skew",
				"channel_02ndSigmaCrossingRate",
				"channel_0MeanCrossingRate",
				"channel_0IQR",
				"channel_075Percentile",
				"channel_0Kurtosis",
				"channel_0Mean"
			],
			"AIF": [
				150,
				133,
				150,
				128,
				109,
				150,
				133,
				126,
				105,
				144,
				150,
				116,
				104,
				150,
				121,
				85,
				87,
				93,
				84,
				89,
				86,
				140,
				140,
				100,
				90,
				150,
				89,
				90,
				150
			],
			"Category": [
				2,
				2,
				1,
				2,
				2,
				1,
				1,
				2,
				2,
				1,
				1,
				1,
				2,
				1,
				1,
				1,
				1,
				1,
				1,
				1,
				1,
				2,
				1,
				2,
				1,
				2,
				1,
				2,
				2
			],
			"Vector": [
				[
					231,
					221,
					138,
					22,
					209,
					180,
					107,
					220,
					25,
					25,
					22,
					138
				],
				[
					224,
					79,
					131,
					22,
					95,
					179,
					97,
					79,
					31,
					30,
					18,
					131
				],
				[
					202,
					15,
					227,
					50,
					20,
					169,
					31,
					17,
					59,
					64,
					28,
					227
				],
				[
					250,
					127,
					138,
					6,
					146,
					180,
					168,
					126,
					5,
					5,
					27,
					138
				],
				[
					253,
					43,
					137,
					8,
					66,
					206,
					56,
					45,
					2,
					2,
					91,
					137
				],
				[
					153,
					13,
					115,
					29,
					41,
					180,
					0,
					13,
					102,
					100,
					1,
					115
				],
				[
					212,
					11,
					130,
					36,
					24,
					157,
					10,
					11,
					45,
					47,
					23,
					130
				],
				[
					251,
					55,
					133,
					23,
					59,
					182,
					178,
					56,
					4,
					3,
					59,
					133
				],
				[
					233,
					17,
					129,
					32,
					40,
					181,
					31,
					17,
					21,
					19,
					69,
					129
				],
				[
					248,
					13,
					107,
					28,
					6,
					195,
					15,
					13,
					6,
					5,
					98,
					107
				],
				[
					239,
					3,
					168,
					37,
					7,
					181,
					5,
					3,
					20,
					24,
					84,
					168
				],
				[
					236,
					15,
					118,
					17,
					22,
					148,
					36,
					14,
					19,
					20,
					40,
					118
				],
				[
					253,
					104,
					137,
					2,
					148,
					176,
					110,
					106,
					2,
					2,
					21,
					137
				],
				[
					219,
					15,
					136,
					115,
					7,
					175,
					25,
					16,
					36,
					35,
					127,
					136
				],
				[
					238,
					10,
					124,
					30,
					9,
					188,
					20,
					9,
					18,
					19,
					48,
					124
				],
				[
					217,
					15,
					143,
					14,
					37,
					176,
					10,
					16,
					37,
					35,
					4,
					143
				],
				[
					233,
					12,
					151,
					20,
					19,
					198,
					46,
					12,
					22,
					21,
					21,
					151
				],
				[
					254,
					17,
					146,
					1,
					35,
					170,
					25,
					17,
					1,
					2,
					13,
					146
				],
				[
					238,
					7,
					162,
					8,
					17,
					174,
					10,
					8,
					17,
					16,
					10,
					162
				],
				[
					232,
					9,
					147,
					11,
					24,
					189,
					13,
					10,
					23,
					24,
					8,
					147
				],
				[
					241,
					12,
					135,
					13,
					27,
					190,
					36,
					13,
					14,
					14,
					20,
					135
				],
				[
					232,
					23,
					136,
					15,
					39,
					182,
					31,
					24,
					23,
					23,
					12,
					136
				],
				[
					251,
					38,
					134,
					3,
					63,
					183,
					36,
					39,
					3,
					3,
					17,
					134
				],
				[
					249,
					67,
					138,
					12,
					88,
					179,
					158,
					67,
					6,
					6,
					36,
					138
				],
				[
					251,
					69,
					136,
					7,
					63,
					183,
					107,
					69,
					4,
					4,
					35,
					136
				],
				[
					253,
					67,
					140,
					15,
					45,
					183,
					138,
					67,
					2,
					2,
					178,
					140
				],
				[
					247,
					65,
					143,
					7,
					88,
					176,
					64,
					66,
					9,
					9,
					19,
					143
				],
				[
					252,
					85,
					137,
					2,
					111,
					180,
					92,
					86,
					3,
					3,
					18,
					137
				],
				[
					253,
					69,
					138,
					2,
					99,
					193,
					117,
					67,
					2,
					2,
					18,
					138
				]
			],
			"Identifiers": [
				0,
				1,
				2,
				3,
				4,
				5,
				6,
				7,
				8,
				9,
				10,
				11,
				12,
				13,
				14,
				15,
				16,
				17,
				18,
				19,
				20,
				21,
				22,
				23,
				24,
				25,
				26,
				27,
				28
			],
			"DistanceMode": 0
		}
	]
}

Similar to previous firmware, first step is to set up our GPIO pin for liquid-atomizer module in src/pincfg_table.c file.

{ // setup Water Atomizer Digital Output
.ucPin = PAD_23,
.ucFunc = PAD23_FUNC_SEL_GPIO_7,
.ucCtrl = PAD_CTRL_SRC_A0,
.ucMode = PAD_MODE_OUTPUT_EN,
.ucPull = PAD_NOPULL,
.ucDrv = PAD_DRV_STRENGTH_4MA,
.ucSpeed = PAD_SLEW_RATE_SLOW,
.ucSmtTrg = PAD_SMT_TRIG_DIS,
},

After setting up GPIO let's modify src/sml_output.c function to direct the action response based upon audio classification or inference performed.

static void sml_output_serial(uint16_t model, uint16_t classification)
{
int count;
int wbytes = 0;
int buflen = sizeof(sensor_ssss_ai_result_buf)-1;
int ret;
kb_get_feature_vector(model, sensor_ssss_ai_fv_arr, &sensor_ssss_ai_fv_len);
// you can send other sensors or GPS data in below line for dashboard
count = snprintf(sensor_ssss_ai_result_buf, buflen,
"{\"ModelNumber\":%d,\"Classification\":%d", (int)model, (int)classification);
wbytes += count;
buflen -= count;

switch((int)classification)
{
// case if unknown detected
case 0:
HAL_GPIO_Write(GPIO_4, 1); //blue LED for unknown
HAL_GPIO_Write(GPIO_5, 0); //green LED off
HAL_GPIO_Write(GPIO_6, 0); //red LED off
break;

// case if 'cricket' detected
case 2:
HAL_GPIO_Write(GPIO_7, 1); // chemical atomizer GPIO 7 on
HAL_DelayUSec(15*1000*1000);  // wait 15 seconds for atomizer
HAL_GPIO_Write(GPIO_7, 0); // chemical atomizer GPIO 7 off
HAL_DelayUSec(15*1000*1000);  // wait 15 before resuming
break;

// case if 'background' detected
case 1:
HAL_GPIO_Write(GPIO_5, 1); //green LED for background
HAL_GPIO_Write(GPIO_4, 0); //blue LED off
HAL_GPIO_Write(GPIO_6, 0); //red LED off
break;
}

After everything is in place, let's compile and flash the firmware to the device using the below command in project directory.

$ make clean
$ make
$ qfprog --port /dev/ttyACM0 --m4app ./qf_ssi_ai_clisensio_pest_app.bin --reset --mode m4

This pest detection and control use case can be improved for detecting killer Hornets and kill those in their breeding seasons before they reproduce in surplus and pose threat to our friend, bees. If pests are not coming near your device, you can put some luring compounds to attract the pests, attraction will lead them to make wingbeat sound near device and the device in turn would activate chemical-atomizer killing those mature pests before they make breeding grounds everywhere.

After, implementing and demonstrating this feature of CliSensio, I got a local mushroom cultivator interested in seeing this device a market ready product to protect his mushroom plantations from white bugs which makes him incur a huge loss, so my next step would be collecting audio data with this device for particular insects or pests and adapt it accordingly for farmers.

Device Power Requirements:

The device runs on solar power and SensiML compute consumes very low power along with microphone, so it can last several months provided, chemical repellent is changed frequently or better in newer version of this Clisensio, I will be adding sensors to alert when to refill using LoRa or cellular.

Also, I am working to add LoRa and Cellular connectivity for the device to extend the notifications and demographic status features (Just the same IoT stuff, sending the data from RX/TX and visualizing on dashboard) but for a POC this is well-understood......More work ahead :)

News Updates & Future Implementations:

We were already troubled and suffering from Covid now "Australia is overrun with its worst plague in decades: millions and millions of mice. The eastern region of New South Wales has been swarmed by an immeasurable number of mice, reports The Associated Press. The mice will cost over half a billion dollars in crop damage, according to the AP." Our CliSensio device can prevent such future outbreaks by detecting mice squeaks and classify their sex, populations in all homes and grain storage houses and eliminate those before the area is swarmed by them using liquid-chemical atomizer module.

The state government has ordered 5,000 liters (1,320 gallons) of the banned poison Bromadiolone from India. The federal government regulator has yet to approve emergency applications to use the poison on the perimeters of crops. Critics fear the poison will kill not only mice but also animals that feed on them. including wedge-tail eagles and family pets.

If you are interested in seeing this device, or swarm of CliSensio devices to solve urgent problems visit us here: https://clisensio.tech/

References:

• https://www.theguardian.com/global-development/2020/may/28/make-noise-and-dont-panic-india-tries-to-ward-off-locust-invasion
• https://patents.google.com/patent/KR100511485B1/en
• https://news.mongabay.com/2021/03/as-climate-change-brings-more-floods-mosquito-numbers-could-swell-study/
• https://www.straitstimes.com/singapore/why-detecting-mosquito-breeding-spots-is-difficult
• https://e360.yale.edu/digest/climate-change-will-expose-half-of-worlds-population-to-disease-spreading-mosquitoes-by-2050
• https://www.wionews.com/science/earth-is-losing-2-bugs-every-year-scientists-call-it-insect-apocalypse-356251/amp
• https://www.wionews.com/india-news/watch-india-suffers-worst-locust-swarm-attack-in-decades-alert-in-delhi-301088
• https://apnews.com/article/plague-ravenous-destructive-mice-tormenting-australians-b8b85ecc13e147f151510e8635d2d032

Clisensio- Not Just Detection but Full Action

I have also shared this article on the PCBWay Community website, https://www.pcbway.com/project/shareproject/CliSensio_Climate_Sensing_and_Insect_Infestation_Control_d7f0c68c.html

Thank you for reading my project :)