Penguin Counting and Monitoring

The Problem

Discarded nets, marine plastic pollution. Sea tempurature rise, marine heatwaves. Coastal errosion and inundation destroy nesting habitat. Predation by feral cats, dogs, mustelids and rats. Things are pretty dire for the Little Blue Penguin (kororā in Maori).

One of the challenges is understanding the impact of these pressures on the population. The main way of measuring the population is to do census in large colonies, check nesting boxes during breeding season (which for kororā can be anywhere from June to March), or to regularly patrol beaches to sample populations - none of these methods really provides a great indication of the population. There have also been some studies where tags have been attached to track individuals.

The plight of Little Blue Penguins (kororā)

Called 'Fairy' penguin in Australia and 'Little Blue' Penguins or the Maori name of 'kororā' in New Zealand. New Zealand's Department of Conservation (DOC) categorise kororā as 'declining/at risk". This species is the smallest of all penguins at just 1kg.

Kororā is present throughout New Zealand and in the south and south east of Australia.

At sea threats include fishing nets and marine plastic pollution that cause choking. Marine heatwaves cause surface sea tempuratures increase, this forces fish into deeper water, this means the penguins must travel further, and dive deeper to find food, and in recent years has lead to starvation and mass deaths. This article shows provides more information on this issue. https://www.nzgeo.com/stories/the-wreck-of-the-penguins/

The problem of marine heatwaves in New Zealand continues to worsen the plight of kororā - https://www.nzherald.co.nz/nz/weather-what-looming-marine-heatwave-means-for-our-ocean-species/GCN6XYY2QK2AJMBG2KRQHOWE2M/

And mass deaths are become more common and worse - https://www.nbcnews.com/news/world/little-blue-penguins-are-washing-dead-new-zealand-beaches-rcna34058

Penguins spend a lot of time at sea, but also return to land often. During nesting and breeding they return to the same area, they, their eggs and chicks are particularly vunerable during this time. Coastal erosion (which has been very very bad in Muriwai this past winter) due to sea level rise, more and higher intensity storms and surges impacts nesting sites.

Land based introduced mamalian predators such as feral cats, dogs (often taken into penguin habitat by owners), cats, mustelids (stoats, weasels, ferrets) and rats eat eggs and kill chicks and adults. In Australia, foxes also pose a serious threat.

The Solution

Part of the solution to these issues is to detect, monitor and protect the penguins when and where they come ashore. Detection is often done by finding occupied burrows or looking for prints in the sand at low tide, however these methods are not scalable, the places where penguins frequent and nest are often remote, and penguins come ashore from dusk, making this task difficult for humans to complete.

The solution is to automate this process using AI and report this using radio (Lora).

The Muriwai Penguin Project

The Muriwai Penguin Project (www.muriwai-environment.org/projects/muriwai-penguin-project) is a group, run by volunteers, to provide monitoring, predator control and trapping, nesting boxes and education and fundraising for penguin rehabilitation and protection.

They take the view that encouraging penguins back into Muriwai's southern bays is a long game, but the team are making progress with kororā starting to nest in some of the 20 boxes deployed. The team used to see predator tracks in the sand at low tide, and after two years of trapping, these tracks started to disappear and the kororā came back to nest in the bays.

How automated monitoring and counting can help?

A series of low cost, long life camera's could help to alert penguin protection groups to the Penguins presence near nesting sites and spark action to deploy more trapping efforts and provide education to locals about their presense and to keep dogs out of the areas.

Potentially the camera's could also be used to detect the presense of predators and threats such as dogs, cats and mustelids (stoats, weasels, ferrets) and alert protectors.

Monitoring numbers coming ashore longer term and over a greater geographic area could contribute to population studies and help to improve protection for the birds.

The Project so far

The project is in no way complete, the challenges still exist (see below), however with the following:

• the POC developed with the K1100 Kit from Seeed, Roboflow and YoloV5
• the low cost, industrial quality AI sensors soon available from Seeed (SenseCAP A1101).
• my visits to the remote southern bays of Muriwai; and
• talking to local experts about the penguin behaviour and the project;

I have no doubt that this project is worthwhile pursuing over the coming New Zealand summer, when access is easier and less impactful on the penguins (eg once chicks have fledged).

Project Challenges

The challenges with this project have been pretty big, mainly due to where the penguins live, and experiencing the wettest New Zealand winter on record.

The penguins of the west coast of Auckland, New Zealand live in very remote and difficult to get to bays. Due to predation and disturbance they are rarely seen in local populated areas (where they were once common place).

Access to these area's is strictly controlled and is via private land, of which the land owners have provided access to the local environmental group. Due to this and the time of year gathering training images has been difficult.

As the penguins are coming ashore to breed and nest at this time of year, extra care needed to be taken so as not to disturb the penguins.

As kororā come ashore from dusk until about midnight, one of the biggest challenges to overcome is to be able to monitor a wide area of beach at night and how to light this area with IR light - while ensuring that the IR light does not disturb the penguins.

This challenge means that I have needed, in the POC, to use a solar set up for IR lighting.

Site Visits

I visited the bays several times during the project and was able to observe penguins in their nests and sitting on eggs. The images below were taken on these visits.

Testing AI models with 'proxy'birds

As it was very difficult to gather training images (until NZ summer), and get into the bays regularly, I decided to gather images on a pond on my property. This pond is frequented by 2 species of duck (Mallard and Paradise) and a New Zealand native rail called a Pūkeko. This would allow me to test models and equipment before later deploying in the difficult to access southern bays.

Also, as one of the challenges of the project would be to distinguish penguins from gulls, this would allow me to test the model to distinguish between species.

While these birds are not noctural (though are quite active at dusk and dawn) and there are obvious distinct differences from kororā, I felt that they would be a 'good enough' proxy to test the AI model, camera, image aquisition and IR lighting.

Training data aquisition

As it is not practical to acquire training data on the Grove AI device or mobile for either penguins or my proxy birds, I am using several methods to capture images.

Firstly I am using a cheap Trail Camera from Kogan (an Australian online retailer) - https://www.dicksmith.co.nz/dn/buy/kogan-hunting-trail-camera-24mp-a-2pk/ - this allows for motion capture, video, images, night vision and timing of video capture - making it suitable - however the motion sensors have a short range.

Secondly I use a great camera that I have set up for use and image capture on Wio Terminal. - https://www.adafruit.com/product/613 please see the code repo attached to this project that shows how I am using this on the Wio Terminal. Here is Adafruits fantastic guide - https://learn.adafruit.com/ttl-serial-camera/wiring-the-camera for the camera and how to use.

Lastly I have aquired images from the web, searching for as many open source images that will be similar to my image captures. All of the images I am using in my Penguin model are currently based on 28 images, I need a lot more.

I also tried using Dall-e AI - https://openai.com/dall-e-2/ - to generate image variations - with actually pretty good results (see image below of the original on the left and 4 variations on right). This helped to provide more training images.

Once I have more images of local penguins I will adjust my internet sourced images to mimic lighting provided on the AI camera with IR lighting.

Solar Power setup

A very handy thing about Little Blue penguins is that they are noctural, only coming ashore in the evening, from dusk until about midnight. This means that devices can be turned off during the day while batteries are charging and switched on during the evening.

As the Wio Terminal does not have great power management, I have opted for a super simple operation for the POC using a 12v timer to switch on and off the battery power to the device and lighting from midnight to dusk. This plugs the devices into the USB ports on the PWM charge controller.

Once the model is deployed using the Seeed - SenseCAP A1101 - LoRaWAN Vision AI Sensor, as it has a long battery life, the solar powered battery will only be required to power IR lighting.

Annotating and Training the Model

I followed the steps in this wiki and it made the set up and training super easy. I used Roboflow to annotate the images. I ended up with just 28 images - all sourced from the internet at this stage.

https://wiki.seeedstudio.com/Train-Deploy-AI-Model-A1101-Grove-Vision-AI/

I was able to acheive very good accuracy with the model - albeit on very limited training data.

mAP - 83.6%

precision - 75.0%

recall - 78.9%

Deploying and running the inference

Again I am using the standard code from this tutorial - https://wiki.seeedstudio.com/Train-Deploy-AI-Model-A1101-Grove-Vision-AI/ and following the steps made the deployment relatively easy. I did find that the model would often fail on load and I would need to reboot or unplug the Wio Terminal to get it to load correctly - I am unsure why this is.

Penguin behaviour allows for quite easy counting, as when they come up the beach the penguins make a b-line for the trees or rocks to get out of danger from gulls and seals. This helps to ensure that we can accurately count the penguins.

There are a number of limitations of Grove AI camera module for my use case.

• Low light - the camera module doesn't have great images in low light;
• IR Filter - if using IR lighting the images are quite washed out;
• FOV - the camera has quite a narrow fov (66 degrees) so not ideal to cover a wider area of the beach;

I was able to infer images of penguins with some accuracy (using test images). I have yet to get valid results of the model on my 'duck proxy' set up.

Lorawan Monitoring via TTN and DataCake

Another challenge is due to the topography and the beaches being remote and at the bottom of cliffs, was to ensure that we would have Lora coverage. The Wio Terminal Lora Field Tester was perfect for this and I was able to check coverage easily.

I am using the Dragino LPS8 Indoor LoRaWAN Gateway with the standard indoor ariel. I am approximately 1.3 km from the edge of the cliff to the beach. Using the Wio Field tester I was able to confirm that coverage would be adequate, but I plan to purchase an outdoor antenna to boost coverage.

I have set up the field tester to push data to TNN and DataCake to get a good picture of coverage in the bays.

I have also set up the Wio Terminal to push data to TTN based on the AI model object detection.

Final test set up

At time of writing I have not deployed the penguin model in my test site and am testing the set up on my pond with my 'proxy' birds. This is not yet fully waterproof so I am deploying only when the weather is good. My model for the proxy birds is not working great with only a couple of detections so far, I assume it is because the camera is in a plastic container so images are not super clear and there is reflection off the water and I don't yet have much training data yet.

I will continue to improve this and collect more training data for the 'stand-in' birds. I have the TTL and Trail cameras also operating to validate the detections. I would say I am some way off deploying a waterproof set up in the penguin test site. There is still work to do on IR lighting (currently supplied by the TTL camera or the Trail cameras if motion sensors are triggered).

Next Steps, enhancements and experiments

As stated the biggest challenge to overcome is to gather more training images and I will be doing this over the New Zealand summer - as of writing (21/9/2022) there are a number of penguins currently on eggs so we will hopefully be able to capture images each day over the coming months as their chicks fledge and the parents go to sea to forage (we need to wait until the eggs are hatched as we don't wish to disturb the birds while they are sitting on eggs).

Other changes and experiments:

• improve proxy bird model and deploy waterproof test set up at my pond.
• investigate changing out the camera module or lens to something like this - https://www.aliexpress.com/item/1005003079676671.html
• more investigation on IR lighting options to optimise how much beach I can cover and still get decent inference.
• investigate using Edge Impulses FOMO model - https://docs.edgeimpulse.com/docs/tutorials/detect-objects-using-fomo - Edge Impulse is releasing support for Grove Vision AI soon.
• look at whether is it possible to deploy the model using the Edge Impulse Himax We I Plus compiler - https://docs.edgeimpulse.com/docs/development-platforms/officially-supported-mcu-targets/himax-we-i-plus and run the inference (I have no idea if this is feasible).

More information on Little Blue Penguins

https://www.doc.govt.nz/globalassets/documents/conservation/marine-and-coastal/marine-protected-areas/monitoring-factsheets/little-penguin-monitoring.pdf

Nest camera in Wellington New Zealand

Photo Credits

https://visuals.newzealand.com/assets/389198 - DunedinNZ

https://visuals.newzealand.com/assets/678356 - Miles Holden