#The Bird Problem in My Rooftop Garden
#My Solution: a tinyML Scarecrow
#Step 1: Making the Scarecrow Smart Enough to Detect Bird Sounds by Training and Deploying an Embedded Machine Learning Model Using Codecraft
1.1 Model Training: on-Board Data Collection & Codecraft Training
1.2 Model Deployment: Graphic Programming the Trained Model to Make the Whole System Works
#Step 2: Build up the Scarecrow with Recycled Materials
#Step 3: Deploy the TinyML Scarecrow in the Rooftop Garden
#Conclusion

Published December 2, 2022 © MIT

TinyML Scarecrow: My Garden Guardian

A tinyML powered scarecrow that triggers buzz to scare away the birds when it detects their sound using Wio Terminal and Codecraft

IntermediateFull instructions provided8 hours545

Things used in this project

Hardware components

Seeed Studio Wio Terminal

Software apps and online services

Edge Impulse Studio

Arduino IDE

Seeed Studio Codecraft

Hand tools and fabrication machines

Scissor, Electrician

Hot glue gun (generic)

Story

I got the chance to own a rooftop garden with some plastic containers as the planting beds since I moved to my current apartment at the beginning of this year. To be honest, I used to be a gardener who only cares about planting beautiful flowers (roses, hydrangea, bougainvillea, viola, jasmine and you name it). After some lockdowns that I encountered, I decided that growing some vegetables can be much more practical and sometimes "life-saving" (lol. Some people found themselves lack of food during lockdowns).

1 / 5

So, here are some pictures of my rooftop garden, which is surrounded by a bunch of trees planted by the landlord. I really love these lush trees as they bring natural shade for my apartment on the top floor, and they also attract lots of birds to stand on top of the trees singing and networking.

Trees and birds everywhere on the rooftop garden

#The Bird Problem in My Rooftop Garden

I never think these cute birds would be a trouble for my garden until I started to sow some bok choy and lettuce in mid October (great time for sowing these veggie here in south China). It turns out the birds love eating these sprouted vegetables so much that they don't want to leave any of them to me. Here are some pictures of the left-over sprouted vegetables.

bok choy and lettuce eaten by the birds

Inspired by a farm scarecrow that I saw in a farm, I decided a scarecrow can give me a hand on scaring away the birds too. However, the birds in the city are so smart and courageous that a traditional scarecrow can't horrify them at all. Usually a traditional scarecrow is dressing like a human and it will only scare the birds when the wind move the sleeves of the scarecrow. As the time goes on, the birds get used to the scarecrow. So I need to think of a smarter and more natural way to trigger the scarecrow.

a traditional scarecrow, credit: www.hgtv.com

#My Solution: a tinyML Scarecrow

Since I have been told how tinyML can bring the machine learning algorithm to edge devices, it's time to build a garden guardian using TinyML. After tinkering around 10 on-and-off days, please meet the TinyML Scarecrow that I built. It's a tinyML powered scarecrow that triggers buzz/voices to scare away the birds when it detects the sound of birds using a Seeed Studio Wio Terminal and Codecraft, a Graphical Programming Platform for Embedded Machine Learning powered by Edge Impulse!

It looks so cute, right? More importantly, it really helps me out by scaring away so many foodie birds by initiating buzz/voices once it detects bird sounds in my rooftop garden. Here are the steps of how I made this tinyML enabled scarecrow.

#Step 1: Making the Scarecrow Smart Enough to Detect Bird Sounds by Training and Deploying an Embedded Machine Learning Model Using Codecraft

In this section, we need to create an embedded machine learning model (voice recognition) using Codecraft platform as the brain of the tinyML Scarecrow to make it smart.

Since there are already some very informative and detailed tutorials on how to set up a Codecraft account, how to connect Wio Terminal to the Codecraft platform, how to train a voice recognition model and deploy the model, I will simplify these steps. For these steps, I'd love to suggest you to check out the following 2 amazing tutorials, which I found really helpful in guiding my learning journey:

Here I'd love to focus more on how this Codecraft platform works to help tinyML beginners to train and deploy an embedded Machine Learning Model to the edge device Wio Terminal.

1.1 Model Training: on-Board Data Collection & Codecraft Training

Simply put, it's the process that you use the Wio Terminal to collect sound data in different labels (in my case, there are 3 labels as below) and sync the collected data on Codecraft to train the model (to let it learn what are these sounds).

BirdAttack: a collection of bird sounds that I recorded on site and I downloaded some Youtube bird sounds of common birds in south China (I want to make sure the scarecrow can detect not only sparrows that are frequent visitors to my garden but also other bird spices just in case )
background: a collection of normal background sounds that I recorded at my rooftop garden (usually it's quite with some noises)
Noise: a collection of sudden noises such as doorbell, children playing noises, cooking, car horn and some other noises that can be heard from the rooftop garden. The reason that I have this third label is that I fInd out the model is much accurate with a separated label of noises. Without this label of data, the model usually detects a noise as a bird sound too.

3 Labels of Voice Data

To make sure I can have enough data to train a qualified and accurate model, I've collect the data of 315s sound in 3 labels (135s for BirdAttack, 90s for background and Noise respectively).

Model Training Report

With these data and neural network//parameters setting, I'm pretty lucky to achieve a 91% accuracy for the model. Here are the parameters that I selected:

neural network size: medium
number of training cycles: 100
learning rate:0.055
minimum confidence rating:0.8

Model Inference Testing: Doing quite well

1.2 Model Deployment: Graphic Programming the Trained Model to Make the Whole System Works

One of the reason that I will recommend the Codecraft platform to beginners lies in it's a graphic programming based on Scratch 3. After getting the model trained, we can click the button "Deploy the Model" to switch to the coding step. The function that I want the tinyML Scarecrow is not complicated and here are codes that I drag and drop:

Final Codecraft Code

What I want the tinyML Scarecrow to do is as follow:

If the confidence of "BirdsAttack" is greater than 0.8 (80%), it will trigger the built-in buzz of Wio Terminal and a custom drew graphic of a yellow bird
If the confidence of "Noise" is greater than 0.8 (80%), it will print a line of Noise in a square on the LCD screen of Wio Terminal
Otherwise, if the confidence of "background" is greater than 0.8 (80%), it will also show the text of Background in a squareon the LCD screen

An important note here. My original plan is to show different images that I designed on the screen but it turns out the images that I designed didn't go well. So, if you want to show a picture on the Wio Terminal LCD screen, you'll need to save the picture in a format of bmp onto a microsd card, and then insert the card to Wio Terminal. For more details about this process, please feel free to follow Seeed Studio's Wiki.

When upload the code to Wio Terminal, here I made a demo video on how it works with bird sound detection.

Model Inference with the final code

If you find the graphics showed on the screen are cute, here I'd love to detail them a bit as I really enjoyed drawing all the graphics using Codecraft. Here is a screenshot of the code with which I drew a scarecrow on the Welcome screen and a bird on the Birds Attack screen.

Codecraft Code of the graphics

Here I also made a small gif on how these graphics shown on the Wio Terminal screen.

graphics on the Wio Terminal Screen

#Step 2: Build up the Scarecrow with Recycled Materials

Once the brain of the tinyML Scarecrow is ready, it's time to build up the "bones" of the guardian. Since I really want to reuse some materials that I have been keeping for some time, I decided to make this scarecrow as sustainable as I can.

1 / 2 • Head of the scarecrow girl

I cut out a huge piece of cloth from an old worn beige t-shirt, filled in some plastic protection nets and sew it into a semi-sphere. By drawing two eyes, a month and sew a nose to the semi-sphere, we now have the head for the scarecrow.

1 / 2

To make it more vivid, I also put a hat on the head and knitted two braids using some recycled macrame cord. Now, we have a head of a scarecrow girl.

As for her body, I crossed 2 plastic parts (length of around 1 meters) in 90 degrees and then used some taps to fix them together. By putting on an upper outer garment donated by my friend, we now have the body for the scarecrow girl.

Scarecrow girl without emotions

When we put the body and the head together, here comes a very cute scarecrow girl.

Scarecrow girl with a smily face

#Step 3: Deploy the TinyML Scarecrow in the Rooftop Garden

After putting the Wio Terminal and the powerbank into a pocket and fixing it onto the scarecrow, the TinyML Scarecrow is ready to start her journey of guarding my rooftop garden! Here I'd love to share some pictures of her standing in the garden.

1 / 4 • TinyML Scarecrow Guarding My Rooftop Garden

#Conclusion

It turns out that Codecraft is really beginner friendly to build your first embedded machine learning model and integrate it to your unique application. For me, I'm really happy to get this tinyML Scarecrow work as my gardening buddy to guardian my garden and to protect my veggie. However, here are some of my future plans to upgrade this project:

1) I haven't figure out how to add a shelter for the TinyML Scarecrow so she can be kept from the rain. Also a reliable power supply is needed too as she should be guarding the garden day and night.

2) I didn't manage to get a MP3 Player module this time as it's out of stock currently. For the upgraded version, I want to record some human voices and conversations, which will be triggered to play randomly when the scarecrow detects birds.

3) Get started with Edge Impulse directly and use Arduino IDE to program the project. This serves as my further learning plan.

Any other suggestions? Please let me know if you have suggestions on the project. Thanks!🤗

Pictures of the veggie guarded by the TinyML Scarecrow

Last but not least, I'd love to express my special thanks to those who came in help during my tinkering:

@Wangli Yang, technical support

@Jane Deng & Alison Yang, video shooting support & debugging

@Jingjing Qiu, upper outer garment supply

@Mr. Potato, thanks for being my tower of strength💚

Code

#include"TFT_eSPI.h"
#include <project_161891_inferencing.h>
enum {ADC_BUF_LEN = 1600};
typedef struct {
  uint16_t btctrl;
  uint16_t btcnt;
  uint32_t srcaddr;
  uint32_t dstaddr;
  uint32_t descaddr;
}dmacdescriptor;
typedef struct {
  signed short *buffers[2];
  unsigned char buf_select;
  unsigned char buf_ready;
  unsigned int buf_count;
  unsigned int n_samples;
}inference_t;
volatile uint8_t recording = 0;
uint16_t adc_buf_0[ADC_BUF_LEN];
uint16_t adc_buf_1[ADC_BUF_LEN];
volatile dmacdescriptor wrb[DMAC_CH_NUM] __attribute__ ((aligned (16)));
dmacdescriptor descriptor_section[DMAC_CH_NUM] __attribute__ ((aligned (16)));
dmacdescriptor descriptor __attribute__ ((aligned (16)));
static inference_t inference;

class FilterBuHp1{
  public:
  FilterBuHp1(){
    v[0] = 0.0;
  }
  private:
  float v[2];
  public:
  float step(float x)
  {
    v[0] = v[1];
    v[1] = (9.621952458291035404e-1f * x) + (0.92439049165820696974f * v[0]);
    return (v[1] - v[0]);
    }
};
FilterBuHp1 filter;

static void audio_rec_callback(uint16_t *buf, uint32_t buf_len) {
  if (recording) {
    for (uint32_t i = 0; i < buf_len; i++) {
      inference.buffers[inference.buf_select][inference.buf_count++] = filter.step(((int16_t)buf[i] - 1024) * 16);
      if (inference.buf_count >= inference.n_samples) {
        inference.buf_select ^= 1;
        inference.buf_count = 0;
        inference.buf_ready = 1;
      }
    }
  }
}

void DMAC_1_Handler() {
  static uint8_t count = 0;
  if (DMAC->Channel[1].CHINTFLAG.bit.SUSP) {
    DMAC->Channel[1].CHCTRLB.reg = DMAC_CHCTRLB_CMD_RESUME;
    DMAC->Channel[1].CHINTFLAG.bit.SUSP = 1;
    if (count) {
      audio_rec_callback(adc_buf_0, ADC_BUF_LEN);
    }else {
      audio_rec_callback(adc_buf_1, ADC_BUF_LEN);
    }
    count = (count + 1) % 2;
  }
}

void config_dma_adc() {
  DMAC->BASEADDR.reg = (uint32_t)descriptor_section;
  DMAC->WRBADDR.reg = (uint32_t)wrb;
  DMAC->CTRL.reg = DMAC_CTRL_DMAENABLE | DMAC_CTRL_LVLEN(0xf);
  DMAC->Channel[1].CHCTRLA.reg = DMAC_CHCTRLA_TRIGSRC(TC5_DMAC_ID_OVF) |
                                 DMAC_CHCTRLA_TRIGACT_BURST;

  descriptor.descaddr = (uint32_t)&descriptor_section[1];
  descriptor.srcaddr = (uint32_t)&ADC1->RESULT.reg;
  descriptor.dstaddr = (uint32_t)adc_buf_0 + sizeof(uint16_t) * ADC_BUF_LEN;
  descriptor.btcnt = ADC_BUF_LEN;
  descriptor.btctrl = DMAC_BTCTRL_BEATSIZE_HWORD |
                      DMAC_BTCTRL_DSTINC |
                      DMAC_BTCTRL_VALID |
                      DMAC_BTCTRL_BLOCKACT_SUSPEND;
  memcpy(&descriptor_section[0], &descriptor, sizeof(descriptor));

  descriptor.descaddr = (uint32_t)&descriptor_section[0];
  descriptor.srcaddr = (uint32_t)&ADC1->RESULT.reg;
  descriptor.dstaddr = (uint32_t)adc_buf_1 + sizeof(uint16_t) * ADC_BUF_LEN;
  descriptor.btcnt = ADC_BUF_LEN;
  descriptor.btctrl = DMAC_BTCTRL_BEATSIZE_HWORD |
                      DMAC_BTCTRL_DSTINC |
                      DMAC_BTCTRL_VALID |
                      DMAC_BTCTRL_BLOCKACT_SUSPEND;
  memcpy(&descriptor_section[1], &descriptor, sizeof(descriptor));

  NVIC_SetPriority(DMAC_1_IRQn, 0);
  NVIC_EnableIRQ(DMAC_1_IRQn);

  DMAC->Channel[1].CHINTENSET.reg = DMAC_CHINTENSET_SUSP;

  ADC1->INPUTCTRL.bit.MUXPOS = ADC_INPUTCTRL_MUXPOS_AIN12_Val;
  while (ADC1->SYNCBUSY.bit.INPUTCTRL);
  ADC1->SAMPCTRL.bit.SAMPLEN = 0x00;
  while (ADC1->SYNCBUSY.bit.SAMPCTRL);
  ADC1->CTRLA.reg = ADC_CTRLA_PRESCALER_DIV128;
  ADC1->CTRLB.reg = ADC_CTRLB_RESSEL_12BIT |
                    ADC_CTRLB_FREERUN;
  while (ADC1->SYNCBUSY.bit.CTRLB);
  ADC1->CTRLA.bit.ENABLE = 1;
  while (ADC1->SYNCBUSY.bit.ENABLE);
  ADC1->SWTRIG.bit.START = 1;
  while (ADC1->SYNCBUSY.bit.SWTRIG);
  DMAC->Channel[1].CHCTRLA.bit.ENABLE = 1;
  GCLK->PCHCTRL[TC5_GCLK_ID].reg = GCLK_PCHCTRL_CHEN |
                                   GCLK_PCHCTRL_GEN_GCLK1;
  TC5->COUNT16.WAVE.reg = TC_WAVE_WAVEGEN_MFRQ;
  TC5->COUNT16.CC[0].reg = 3000 - 1;
  while (TC5->COUNT16.SYNCBUSY.bit.CC0);
  TC5->COUNT16.CTRLA.bit.ENABLE = 1;
  while (TC5->COUNT16.SYNCBUSY.bit.ENABLE);
}

static bool microphone_inference_record(void) {
  bool ret = true;
  while (inference.buf_ready == 0) {
    delay(1);
  }
  inference.buf_ready = 0;
  return ret;
}

static int microphone_audio_signal_get_data(size_t offset,
                                              size_t length,
                                              float *out_ptr) {
  numpy::int16_to_float(&inference.buffers[inference.buf_select ^ 1][offset], out_ptr, length);
  return 0;
}

TFT_eSPI tft;
ei_impulse_result_classification_t currentClassification[EI_CLASSIFIER_LABEL_COUNT];
const char* maxConfidenceLabel;

void runClassifier()
{
  bool m = microphone_inference_record();
  if (!m) {
    return;
  }

  signal_t signal;
  signal.total_length = EI_CLASSIFIER_SLICE_SIZE;
  signal.get_data = &microphone_audio_signal_get_data;
  ei_impulse_result_t result = { 0 };
  EI_IMPULSE_ERROR r = run_classifier_continuous(&signal, &result, false);
  if (r != EI_IMPULSE_OK) {
    return;
  }

  float maxValue = 0;
  for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
    ei_impulse_result_classification_t classification_t = result.classification[ix];
    ei_printf("    %s: %.5f\n", classification_t.label, classification_t.value);
    float value = classification_t.value;
    if (value > maxValue) {
      maxValue = value;
      maxConfidenceLabel = classification_t.label;
    }
    currentClassification[ix] = classification_t;
  }
}
float getLabelConfidence(char *label)
{
  float value = 0;
  for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
    ei_impulse_result_classification_t classification = currentClassification[ix];
    if (label == classification.label) {
      value = classification.value;
    }
  }
  return value;
}

void setup(){
  tft.begin();
  run_classifier_init();
  inference.buffers[0] = (int16_t *)malloc(EI_CLASSIFIER_SLICE_SIZE * sizeof(int16_t));
  if (inference.buffers[0] == NULL) {
    return;
  }
  inference.buffers[1] = (int16_t *)malloc(EI_CLASSIFIER_SLICE_SIZE * sizeof(int16_t));
  if (inference.buffers[1] == NULL) {
    free(inference.buffers[0]);
    return;
  }

  inference.buf_select = 0;
  inference.buf_count = 0;
  inference.n_samples = EI_CLASSIFIER_SLICE_SIZE;
  inference.buf_ready = 0;

  config_dma_adc();

  recording = 1;

  pinMode(WIO_BUZZER, OUTPUT);
  tft.setRotation(3);
  tft.fillScreen(0xFD60);
  tft.setTextSize(2);
  tft.setTextColor(0xFFFF);
  tft.drawTriangle(60,150,8,60,75,60,0xF800);
  tft.fillTriangle(60,150,8,60,75,60,0xF800);
  tft.drawCircle(40,40,20,0xFDD7);
  tft.fillCircle(40,40,20,0xFDD7);
  tft.drawCircle(35,35,3,0x0);
  tft.fillCircle(35,35,3,0x0);
  tft.drawCircle(50,35,3,0x0);
  tft.fillCircle(50,35,3,0x0);
  tft.drawRoundRect(40,45,10,5,1,0xFD75);
  tft.fillRoundRect(40,45,10,5,1,0xFD75);
  tft.drawTriangle(35,10,20,25,60,25,0x101F);
  tft.fillTriangle(35,10,20,25,60,25,0x101F);
  tft.drawString((String)"tinyML Scarecrow", 100, 100);
  tft.drawRoundRect(90,90,210,40,5,0xFFFF);
  delay(5000);

}



void loop(){

  runClassifier();
  if ((getLabelConfidence("BirdAttack") > 0.8)) {
    tone(WIO_BUZZER,698);
    delay(1000);
    noTone(WIO_BUZZER);
    tft.fillScreen(0xF800);
    tft.drawRoundRect(20,120,100,100,7,0xFE80);
    tft.fillRoundRect(20,120,100,100,7,0xFE80);
    tft.drawTriangle(20,125,5,150,25,150,0xFFFF);
    tft.fillTriangle(20,125,5,150,25,150,0xFFFF);
    tft.drawCircle(40,140,5,0x0);
    tft.fillCircle(40,140,5,0x0);
    tft.drawCircle(43,141,2,0xFFFF);
    tft.fillCircle(43,141,2,0xFFFF);
    tft.drawTriangle(70,180,80,160,100,190,0x1F);
    tft.fillTriangle(70,180,80,160,100,190,0x1F);
    tft.drawTriangle(120,120,130,110,140,130,0x1DF);
    tft.fillTriangle(120,120,130,110,140,130,0x1DF);
    tft.drawString((String)"Birds Attack", 100, 60);
    tft.drawRoundRect(90,50,160,40,5,0xFFFF);
    delay(1000);
  } else {
    if ((getLabelConfidence("Noise") > 0.8)) {
      tft.fillScreen(0x1D80);
      tft.drawString((String)"Noise", 100, 100);
      tft.drawRoundRect(70,90,100,40,5,0xFFFF);
      delay(1000);
    } else {
      tft.fillScreen(0x588);
      tft.drawString((String)"Background", 100, 100);
      tft.drawRoundRect(85,90,150,40,5,0xFFFF);
      delay(1000);
    }
  }

}