Smart Fishing Rod with M5StickC

When I tried throwing fishing for the first time in a while, it was over, but I had to watch the movement of the rod tip.

I haven't seen it before and a big one hit it and the pole jumped into the sea.

At that time, my friend miraculously picked up from the sea with a lure

It was quite troublesome to look at the rod tip and wait for Atari.

So when I thought about how I could do it electronically, discover M5StickC!

I bought it immediately at Switch Science and tried to build a waiting system.

As a strategy, attach an acceleration sensor (weight 4 g) connected to the rod tip with a Grove cable

We plan to send that value via Bluetooth with M5 StickC, receive it on a PC and plot a graph of acceleration.

It's also attractive because it has an Atom (weight 14g) without a screen and an accelerometer.

For the time being, I decided to choose a combination of M5 Stick and an external sensor that I could play with.

There seems to be a new M5 Stick C+ with a larger battery, but since the stock was 0, it was made an old model.

By the way, I think I can develop with just this, but it seems that I can actually do it.

The M5 Stick C is a great product that comes in a lot and is ready to use.

About M5StickC

When I touched the RaspberryPi when I was v1, I could understand it.

I knew only about the name of Arduino

I used to have to make a PIC from a writer in the old days

I was frustrated on the way,

Arduino seems to be able to enter the development environment immediately for free

I think that this can be done, and when I try to put in an IDE

Moreover, it seems that you can write immediately if you connect with USB

It's a good time. With this, you can use it immediately.

That's why I installed the IDE immediately

Referring to the articles of excellent ancestors

When the power was turned on, a switch science demo sketch that showed tilts moved.

Let's connect them now. It is completed when connected with the Grove cable.

It's too easy and I miss it.

I used it because the sensor had a short cable.

In actual battle, we plan to connect the sensor to the sensor at the rod tip using the longest cable of 2 m.

The axis direction is printed on the ADXL345 unit, which I learned later, but it's very easy to understand when developing.

For a while after I arrived, I learned a lot from Qiita's articles on how to use the IDE and how to write.

Since various people make it in the same environment, it is very fun to use simple things just by copying sketches.

Anyway, it basically works if connected.

Now that I've learned how to use Bluetooth and serial communication, I have coded.

All you have to do is skip the sensor data to Bluetooth.

Code on the fishing rod side

As a strategy, the fishing rod side needs to transmit the value of the acceleration sensor at high speed.

First of all, the program was limited to the simple sending function.

On the screen, simply the root of the square of the acceleration in the x, y, zx, y, z direction (length of the three-dimensional vector), that is, the following formula

Only the magnitude of the acceleration vector calculated by is displayed.

When there is no acceleration, the value of 1g of the earth's acceleration is displayed.

In other words, without limiting the moving direction, if you move in any direction, this AA value will deviate from 1g.

This time, let's assume that this value is used for the judgment.

I'll add other features later if I can afford the processing power.

Library used

There seems to be some libraries of ADXL345, but this time I searched with Arduino IDE and it came out

https://www.arduinolibraries.info/libraries/accelerometer-adxl345

I used.

I didn't use it this time, but there are other

It seems to have various built-in functions such as touch detection and free fall detection.

It seems to be easy to use, so it seems to have a wide range of applications.

Completed sketch

That's why the completed M5 Stick C side code is

surido.ino
/*****************************************************************************/
//  Function:    Get the accelemeter of X/Y/Z axis and print out on the 
//                  serial monitor and bluetooth.
//  Usage:       This program is for fishing. Use the accelerometer at the end
//               of the rod to see if the fish is caught. Acceleration is 
//               transmitted in real time via Bluetooth and can be monitored 
//               from a laptop.
//  Hardware:    M5StickC + ADXL345(Grove)
//  Arduino IDE: Arduino-1.8.13
//  Author:  Hideto Manjo     
//  Date:    Aug 9, 2020
//  Version: v0.1
//
//  This program is free software; you can redistribute it and/or
//  modify it under the terms of the GNU Lesser General Public
//  License as published by the Free Software Foundation; either
//  version 2.1 of the License, or (at your option) any later version.
//  This library is distributed in the hope that it will be useful,
//  but WITHOUT ANY WARRANTY; without even the implied warranty of
//  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
//  Lesser General Public License for more details.
//  You should have received a copy of the GNU Lesser General Public
//  License along with this library; if not, write to the Free Software
//  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
//
/*******************************************************************************/
#include <M5StickC.h>
#include <BLEDevice.h>
#include <BLEServer.h>
#include <BLE2902.h>
#include <ADXL345.h>
#include <Wire.h>

#define SERVICE_UUID        "8da64251-bc69-4312-9c78-cbfc45cd56ff"
#define CHARACTERISTIC_UUID "deb894ea-987c-4339-ab49-2393bcc6ad26"
#define DEVICE_NAME         "Tsurido"
#define LCD_ROTATION        0      // Screen rotation angle = 90 * LCD_ROTATION, Specify 2 when 180 degrees [counterclockwise]
#define DELAY               50     // specified in microseconds
#define SERIAL              true   // Presence of serial output (USB)
#define BAUDRATE            115200 // Serial output baud rate setting

#define SCALAR(x, y, z)     sqrt(x*x + y*y + z*z)

BLEServer* pServer = NULL;
BLECharacteristic* pCharacteristic = NULL;
ADXL345 adxl;

// true if there is a ble connection
bool deviceConnected = false;

int x = 0;
int y = 0;
int z = 0;
int scalar;
char msg[128];


class MyServerCallbacks: public BLEServerCallbacks {
                void onConnect(BLEServer* pServer) {
                        M5.Lcd.setCursor(10, 110);
                        M5.Lcd.println("Con");
                        deviceConnected = true;
                }

                void onDisconnect(BLEServer* pServer) {
                        M5.Lcd.setCursor(10, 110);
                        M5.Lcd.println("   ");
                        deviceConnected = false;
                }
};

void setup_adxl345() {
        adxl.powerOn();
}

void setup_ble(){
        BLEDevice::init(DEVICE_NAME);
        BLEServer *pServer = BLEDevice::createServer();
        pServer->setCallbacks(new MyServerCallbacks());
        BLEService *pService = pServer->createService(SERVICE_UUID);
        pCharacteristic = pService->createCharacteristic(
                                  CHARACTERISTIC_UUID,
                                  BLECharacteristic::PROPERTY_READ |
                                  BLECharacteristic::PROPERTY_WRITE |
                                  BLECharacteristic::PROPERTY_NOTIFY |
                                  BLECharacteristic::PROPERTY_INDICATE
                          );
        pCharacteristic->addDescriptor(new BLE2902());

        pService->start();
        BLEAdvertising *pAdvertising = pServer->getAdvertising();
        pAdvertising->start();
}

void setup() {
        M5.begin();

       // Confirm pin assignment of M5StickC SDA=32, SDL=33
         // It is printed on the place that points to the Grove cable //
        Wire.begin(32, 33);

        Serial.begin(BAUDRATE);
        setup_adxl345();
        setup_ble();
        M5.Lcd.setRotation(LCD_ROTATION);
        M5.Lcd.fillScreen(BLACK);
        M5.Lcd.setTextSize(2);
        M5.Lcd.setCursor(10, 0);
        M5.Lcd.println("Tsuri");
        M5.Lcd.println("  do ");
}

void loop() {
        M5.update();

        // Read acceleration data from ADXL345
        adxl.readXYZ(&x, &y, &z);

        // Convert to vector size (scalar amount)
        scalar = SCALAR(x, y, z);
        M5.Lcd.setCursor(0, 60);
        M5.Lcd.println("Acc");
        M5.Lcd.printf("  %d\n", scalar);
        sprintf(msg, "Ax, Ay, Az, A: %d, %d, %d, %d", x, y, z, scalar);

        if (SERIAL)
            Serial.println(msg);

        if (deviceConnected) {
                pCharacteristic->setValue(msg);
                pCharacteristic->notify();
        }

        delay(DELAY);
}

When you start up, it will be the first time to measure acceleration, if you connect from bluetooth

"Ax, Ay, Az, A: 2, 2, 255, 16"

Send a string every 50ms in the same format as a serial plotter.

In this example, about 1g1g is measured in the Az direction.

Since it is not necessary to measure the precise value of acceleration, the adjustment of gain etc. is ignored and the integer value of the sensor is transmitted as it is this time.

Laptop side (client) production

Next, let's suppose you make a laptop computer.First, create a string from bluetooth.

Perth function

@staticmethod
def _parse(data):
    # First: divide by position
    labels, values, = str(data).split(":")
    After splitting with #, strip the leading and trailing spaces
    labels = [label.strip() for label in labels.split(',')]
    values = [value.strip() for value in values.split(',')]

   Align # labels to the same length as values
    if len(labels) > len(values):
        labels = labels[0:len(values)]
    else:
        labels += (len(values) - len(labels)) * [""]

    return labels, values

As an example

"Ax, Ay, Az, A: 2, 2, 255, 255"

When a character string like

Return value of parse function
labels = ["Ax", "Ay", "Az", "A"]
values = ["2", "2", "255","16"]

Returns two lists of.

Even if it is a serial plotter, it seems to work without a label

If the lengths are different, we try to match the values.

Atari Judgment

To judge Atari from sensor data

I was trying to judge by the threshold value that was set manually.

If you think carefully, there is also the influence of the wind, so I thought about it a little more

I came up with the idea of finding outliers.

In other words, calculate the average value and standard deviation of the sensor data

For example, if it deviates from the average value by 2σ or more, it feels like it is atari.

numpy has functions for both mean and standard deviation, so it was easy to create.

#First calculate mean and standard deviation
mean = self.y.mean()
std = self.y.std()

# Only the amount of deviation from the average value
#Plotting this diff, the lowest value is always 0 and easy to use
diff = np.abs(self.y - mean)

# Pass the amount of deviation and standard deviation to the Atari judgment function
self._check_warning(diff, std)

The hit judgment function is as follows.

What is overrange is also the upper limit of the vertical axis of the graph,

This is because it is decided based on this standard deviation.

# self.sigma = [3, 5]
def _check_warning(self, diff, std):
    if time.time() - self.last_ring > 2:
        if diff[-1] > self.sigma[1] * std:
            # When overranged
            self.last_ring = time.time()
            SoundPlayer.play("warning2.mp3")
        elif diff[-1] > self.sigma[0] * std:
            # アタリ
            self.last_ring = time.time()
            SoundPlayer.play("warning1.mp3")

last_ring is the time when the hit was judged last time, and it waits for 2 seconds until the next judgment.

Judgment test

For the time being, the display range of the graph is 5σ

The threshold value for Atari is set to 3σ.

I am moving the sensor by hand, and a sound will be emitted when the yellow line (3σ) is crossed and when the range is over (5σ).

Sensitivity adjustment is also calculated automatically based on the standard deviation, so it seems that you can judge it quite nicely.

(The image is in the process of making a client. The accelerometer is shaken by hand around 1000 counts.

Then, the acceleration deviation exceeds the 3σ line and a sound is produced. )

So it seems that I can manage it.

Other details

Here are some things that made me interested in making a client. If you are in a hurry, skip to the next chapter, as it is not an important part.

Make plot window exit when closing the plot window

Even if you close the window if the part received by the sub thread is alive

It is generated indefinitely and cannot be terminated.

So I made it possible to end together.

Judgment part when the plot window is closed
from matplotlib import _pylab_helpers

if _pylab_helpers.Gcf.get_active() is None:
    # Raise closed flag when window is closed
    self.closed = True
    return

The part that gets out of the bluetooth loop after being judged
Continue to sleep only when the closed flag of #plotter is false.
while not PLOTTER.closed:
    time.sleep(0.1)

print('Disconnecting to device...')
device.disconnect()

print("Stop.")
return 0

Sleeping while judging the PLOTTER.closed flag.

The bluetooth thread is running in a subthread

If you are waiting forever, Ctrl-C won't stop and it was hard.

This is a temporary solution.

By doing this, if you close windows for the time being, you can exit by device.disconnect() 0.1 seconds later.

Plot matplotlib in background

If I plot with pause, I can plot, butThe window becomes active.I looked up there and found a solution, so I fixed the code.I have searched quite a bit, so I will write it down.

plt.pause(self._pause)

This will make the window always active

Replaced code
plt.ion()

plt.gcf().canvas.draw_idle()
plt.gcf().canvas.start_event_loop(self._pause)

This way it can be displayed in the background

Calculation of the angle of the rod tip

In any case, if you also calculate the angle of the rod tip

It's likely to be prepared even if the rod bends a little after eating fish

I also calculated the tilt angle from the acceleration.

Calculation method is AN-1057: Tilt detection by accelerometer-Analog Devices

I was riding https://www.analog.com/media/jp/technical-documentation/application-notes/AN-1057_jp.pdf

According to it, the value of the accelerometer is as follows

It seems that each angle can be obtained by converting to polar coordinates. Please refer to the link above to see which part these two angles represent.

It should be noted that the unit on the left side at this time is radian, so it is necessary to multiply by 180/π180/π to convert it to an angle.

If the xx axis direction of the ADXL345 with Goove connection is aligned with the direction of the rod, then θθ is the angle of the rod tip.

Note that θθ changes its sign depending on whether the yy axis of ADXL345 is oriented in the direction of gravity or vice versa.

Although it takes a negative value, only the angle with the horizontal plane is needed this time, so when using the calculation result, the absolute value is taken with np.abs.

The calculation is branched by an if statement so that it will not divide into zeros and cause an error.

Even if the size is 0, 0 is returned so that ϕϕ does not cause an error.

Calculation of the angle of the rod tip
@staticmethod
def angle(ax, ay, az):
     When # ay is 0, it is divided by 0 and it becomes an indeterminate form, so assign a fixed value
     if ay != 0:
         theta = np.arctan(ax / ay) * 180 / np.pi
     else:
         if ax> 0:
             theta = 90
         elif ax <0:
             theta = -90
         else:
             theta = 0

     In the case of # phi, if the denominator becomes zero, do not calculate and substitute a substitute value
     scalor = np.sqrt(ax*ax + ay*ay + az*az)
     if scalor != 0:
         phi = np.arccos(az / scalor) * 180 / np.pi
     else:
         phi = 0

     return theta, phi

Completion of code on laptop

Here are the twists and turns

""" Trurido client

Copyright 2020 Hideto Manjo.
Licence: LGPL
"""

import time
import uuid
import numpy as np

from matplotlib import pyplot as plt
from matplotlib import _pylab_helpers

from pydub import AudioSegment
import simpleaudio

import Adafruit_BluefruitLE

plt.style.use('dark_background')
plt.rcParams['toolbar'] = 'None' 

SERVICE_UUID = uuid.UUID("8da64251-bc69-4312-9c78-cbfc45cd56ff")
CHAR_UUID    = uuid.UUID("deb894ea-987c-4339-ab49-2393bcc6ad26")


class SoundPlayer:
    """SoundPlayer module."""

    @classmethod
    def play(cls, filename, audio_format="mp3", wait=False, stop=False):
        """Play audio file."""

        if stop:
            simpleaudio.stop_all()

        seg = AudioSegment.from_file(filename, audio_format)
        playback = simpleaudio.play_buffer(
            seg.raw_data,
            num_channels=seg.channels,
            bytes_per_sample=seg.sample_width,
            sample_rate=seg.frame_rate
        )

        if wait:
            playback.wait_done()


class Plotter:
    def __init__(self, interval=1, width=200, pause=0.005, sigma=(5, 7),
                 angle=True, xlabel=False, ylabel=True):
        # field length
        self.__fieldlength = 4

        # main plot config
        self._target_index = 3
        self._interval = interval
        self._pause = pause
        self._width = width
        self._angle = angle
        self.sigma = sigma   # [warning, overrange]

        self.count = 0
        self.closed = False # plotter end flag
        self.last_ring = time.time() + 3

        self.t = np.zeros(self._width)
        self.values = np.zeros((self.__fieldlength, self._width))

        # initial plot
        plt.ion()
        self.fig = plt.figure()
        self.fig.canvas.set_window_title('Tsurido Plotter')
        if self._angle:
            self.ax1 = self.fig.add_subplot(2, 1, 1)
        else:
            self.ax1 = self.fig.add_subplot(1, 1, 1)

        self.li, = self.ax1.plot(self.t, self.values[self._target_index],
                                 label="Acc", color="c")
        self.li_sigma = self.ax1.axhline(y=0)

        if xlabel:
            plt.xlabel("count")

        if ylabel:
            self.ax1.set_ylabel(self.li.get_label())

        if angle:
            self.tip_angle = np.zeros(self._width)
            # self.ax2 = self.ax1.twinx()
            self.ax2 = self.fig.add_subplot(2, 1, 2)
            self.li2, = self.ax2.plot(self.t, self.tip_angle,
                                      label="Rod angle", color="r")
            self.ax2.set_ylabel(self.li2.get_label())

    @staticmethod
    def _parse(data):
        labels, values, = str(data).split(":")
        labels = [label.strip() for label in labels.split(',')]
        values = [value.strip() for value in values.split(',')]

        # Make the lists of labels and values the same length, fill in the blanks
        if len(labels) > len(values):
            labels = labels[0:len(values)]
        else:
            labels += (len(values) - len(labels)) * [""]

        return labels, values

    @staticmethod
    def angle(ax, ay, az):
        if ay != 0:
            theta = np.arctan(ax / ay) * 180 / np.pi
        else:
            if ax > 0:
                theta = 90
            elif ax < 0:
                theta = -90
            else:
                theta = 0

        phi = np.arccos(az / np.sqrt(ax*ax + ay*ay + az*az)) * 180 / np.pi

        return theta, phi

    def _store_values(self, values):
        self.t[0:-1] = self.t[1:]
        self.t[-1] = float(self.count)
        self.count += 1

        self.values[:, 0:-1] = self.values[:, 1:]
        self.values[:, -1] = [float(v) for v in values]

    def _store_angle(self, values):
        angle = self.angle(*[float(v) for v in values[:3]])
        self.tip_angle[0:-1] = self.tip_angle[1:]
        self.tip_angle[-1] = np.abs(angle[0])

    def _check_warning(self, diff, std):
        if time.time() - self.last_ring > 2:
            if diff[-1] > self.sigma[1] * std:
                # over range
                self.last_ring = time.time()
                SoundPlayer.play("sfx/warning2.mp3")
            elif diff[-1] > self.sigma[0] * std:
                # warning
                self.last_ring = time.time()
                SoundPlayer.play("sfx/warning1.mp3")

    def _plot(self, y, std):
        self.li.set_xdata(self.t)
        self.li.set_ydata(y)

        self.li_sigma.set_ydata([self.sigma[0] * std, self.sigma[0] * std])

        self.ax1.set_ylim(0, self.sigma[1] * std)
        if self.count != 1:
            self.ax1.set_xlim(np.min(self.t), np.max(self.t))

        if self._angle:
            self.li2.set_xdata(self.t)
            self.li2.set_ydata(self.tip_angle)
            self.ax2.set_xlim(np.min(self.t), np.max(self.t))
            self.ax2.set_ylim(self.tip_angle.mean() - 10,
                              self.tip_angle.mean() + 10)

        plt.gcf().canvas.draw_idle()
        plt.gcf().canvas.start_event_loop(self._pause)

    def received(self, data):
        if _pylab_helpers.Gcf.get_active() is None:
            # end flag
            self.closed = True
            return

        # Parse
        _labels, values = self._parse(data)

        # Store value as new values
        self._store_values(values)

        # Calculate and store rod tip angle
        if self._angle:
            self._store_angle(values)

        # Calculate standard deviation
        y = self.values[self._target_index]
        std = y.std()

        # Calculate deviation from average value
        diff = np.abs(y - y.mean())

        # Sounds a warning when the deviation exceeds a specified value
        self._check_warning(diff, std)

        # Since plotting is slow, accurate graphs can be drawn by
        # thinning out the display.
        if self.count % self._interval == 0:
            self._plot(diff, std)

        # print('{0}'.format(data))

def main():
    BLE.clear_cached_data()

    adapter = BLE.get_default_adapter()
    adapter.power_on()
    print('Using adapter: {0}'.format(adapter.name))

    print('Disconnecting any connected devices...')
    BLE.disconnect_devices([SERVICE_UUID])

    print('Searching device...')
    adapter.start_scan()

    try:
        device = BLE.find_device(name="Tsurido", timeout_sec=5)
        if device is None:
            raise RuntimeError('Failed to find device!')
    finally:
        adapter.stop_scan()

    print('Connecting to device...')
    device.connect()

    print('Discovering services...')
    device.discover([SERVICE_UUID], [CHAR_UUID])

    uart = device.find_service(SERVICE_UUID)
    chara = uart.find_characteristic(CHAR_UUID)

    print('Subscribing to characteristic changes...')
    chara.start_notify(PLOTTER.received)

    # wait
    while not PLOTTER.closed:
        time.sleep(0.1)

    print('Disconnecting to device...')
    device.disconnect()

    print("Stop.")
    return 0


if __name__ == '__main__':
    BLE = Adafruit_BluefruitLE.get_provider()
    PLOTTER = Plotter(interval=4, angle=True)

    BLE.initialize()
    BLE.run_mainloop_with(main)

Operation scenery

Eventually we have a client like the one below.

(The acceleration deviation is displayed above and the rod tip angle is displayed below. If the acceleration deviation exceeds the yellow line, a collision judgment is made and a sound is issued. The above example is shaken by hand.)

It's unclear what happens when you actually catch a fish, but I'm looking forward to this.

Attach to the pole

For the rod, use the vinyl tape for screwing the M5 Stick C.

I attached it to the rod.

(Figure set on the rod)

I'm worried about the internal battery of the M5 Stick C, so use a USB cable

I have installed a 500mA battery from a 100mA shop at 4000mAh.

You can use it quite well.

The sensor was also attached with vinyl tape. I'm worried about flying to the sea.

Start fishing

The truth is that if you don't set the angle of the fishing rod at a higher angle, it may be pulled by the fish and fall into the sea.

This time, I had no belongings, so I leaned against a folding chair. I am worried because the angle is a little shallow.

Well, can you really fish...

Set the rod and wait for Atari in the cooler car.

Watch the laptop in the car with the rod set

This is the first time to fish while watching the monitor in the car.

I really caught

When I'm preparing to shoot a video...

There is a beeping sound. ! !

Atari.

If you look at the rod, thinking that it might be the effect of the wind...

I was really talking.

If you hurry up and pull up...

It was about 15 cm.

No, it's bad. In about two minutes after throwing in the rod

Atari sounds exactly as I expected, and I actually hit and caught the fish.

Unfortunately I didn't leave a log of the graph at the moment I caught it, so

Although there are no records of the first moving Atari

You can fish with this system.

Moreover, when you look at the graph

It seems that you can see more detailed atari than visually.

This really works. ! !

Probably a great success, Atari judgment has already reached a practical level.

What kind of waveform will appear when actually hit

After setting the camera etc, I could not actually fish

There was some Atari.

It is the situation at that time. The image below is a composite of plots and the like in OBS similar to the image of the game called fishing road.

Water temperature is not the value of the sensor, but it may be possible to measure it in real time by using M5StickC.

Sorry for the slightly hard-to-see image, but there is a Plotter plot in the lower right,

You can see that the acceleration deviation exceeds the threshold and reacts.

The lower inclinometer vibrates periodically like the waveform of an earthquake.

I didn't think it would be so clear.

You may also be able to distinguish the type of fish and the timing of matching.