Published July 22, 2021 © CC BY-NC

bAIwatch, deep learning and surf

bAIwatch gets images from a public webcam from a beach, evaluates the surfing conditions and warns me it the waves are good enough.

AdvancedFull instructions provided689

Things used in this project

Hardware components

Raspberry Pi 3 Model B

Linear Solenoid, 12 VDC

Digilent Voltage Regulator Module

Seeed Studio Grove - 2-Channel SPDT Relay

Software apps and online services

Raspberry Pi Raspbian

Google Colaboratory

TensorFlow

Autodesk Fusion

Hand tools and fabrication machines

3D Printer (generic)

Story

With english subtitles.

Hi!

In this tutorial I’m going to show you how I made the bAIwatch. This device is capable of being connected to the webcam located in a beache where I usually go surfing, while it evaluates the state of the waves and lets me know, thanks to the vibration of a solenoid, when the conditions are good to leave everything, take the board and go surfing!

Step 1: Downloading Images.

This project is based on an image classifier. The first step was collecting a good amount of pictures of the spot, with the aim of gathering images of every possible conditions: sunny days, cloudy days, heavy rain days, soft rain days...

1 / 3 • Sunny day.

For this, I used the notebook below (pictureDownloader on Code section) on Google Colab (I’ve changed the target webcam url of the script). In this case, I'm using urllib to download the.jpg files directly to Google Drive -for other webcam configurations you may have to use Selenium-.

Step 2: Preparing the Data

After collecting data, I made another script (imageClassificator on Code section) to make a manual classification for preparing the dataset.

I made three categories:

1- Good waves, with all the images with acceptable waves.

2- Bad waves, with both photos of calm and shattered seas impossible to surf for me.

3- Discards, with all the photos that don't point to the place of the beach that I'm trying to analyze (because they point to another place on the beach, the parking lot... whatever).

1 / 3 • Too big waves.

Step 3: Evaluating the CNN

Once I had all the downloaded images on those 3 categories I started training the AI. For this, I used this notebook as a guide:

https://www.kaggle.com/vincee/intel-image-classification-cnn-keras

The result was quite good, with 84% accuracy. Here you have the confusion matrix, where you can see the result in a more graphic way:

Step 4: Assembling It All.

1 / 2 • Cool wave box.

After that, I just put it inside a raspberry pi 3. Here you can find a guide to install tensorflow 2.2 on it.

https://github.com/sagarrabanana/Install-OpenCV-and-TF-2.2.0-on-RPi-3

Remember that you must downgrade the version of tensorflow on Google Colab before training the AI.

Here you have the adapted code I made for the raspberry (mainScript on Code section). You must configure the RP to run this script automatically when the RP is turned on.

I also made a cool box and assembled everything inside.

Here you have the schema.

Schema

I hope you like the project! This may not be the best documentation, but I think it can be very handy if you are trying to replicate it.

Of course, you can contact me for further information!

Code

from google.colab import drive
drive.mount('/content/gdrive')

import time
import datetime
import urllib.request
import socket
import matplotlib.pyplot as plt
socket.setdefaulttimeout(30)



#while True:
while (int(datetime.datetime.now().strftime("%H"))>5 and int(datetime.datetime.now().strftime("%H"))<17): 
  # setting filename and image URL
  try:
    print('dentro del try')
    name=str(datetime.datetime.now())
    filename = '/content/gdrive/MyDrive/Dataset/'+name+'.jpg' #here goes the path to your dataset folder on your google drive
    image_url = "http://212.8.113.121:8081/jpg/1/image.jpg"
    urllib.request.urlretrieve(image_url, filename)
    print('imagen '+name+' descargada.')

    #####
    imgBig=image.load_img(filename)
    plt.imshow(imgBig)
    plt.show()


    ####


    print('waiting...')
    time.sleep(30)
  except:
    print('try again in 30 sec')
    time.sleep(30)

# this script is made to help in the manual classificatio. You will have to modify the directories before using.
# the script will show you all the images from the main directory and move them to their correct place by pressing B(for good waves), M(for bad ones) and D(for discards)

import os
import shutil
import cv2
import time
i=0
directory = r'dataset directory'#'/home/sagarrabanana/Escritorio/dataSet_Plentzia/SanValentin'
directorySaveBuenas = r'folder for good waves'
directorySaveMalas = r'folder for bad waves'
directorySaveDescarte = r'folder for discarted images'
for filename in sorted(os.listdir(directory)):
    cv2.destroyAllWindows()
    if filename.endswith(".jpg") or filename.endswith(".png"):
        img=cv2.imread(os.path.join(directory, filename))
        cv2.imshow(filename,img)
        key=cv2.waitKey(0)
        if key== ord('b'):
            print('tecla B pulsada')
            shutil.move(directory+'/'+ filename,directorySaveBuenas+'/'+filename)
            print('movida a Buenas')
        elif key== ord('m'):
            print('tecla M pulsada')
            shutil.move(directory+'/'+ filename,directorySaveMalas+'/'+filename)
            print('movida a Malas')
        elif key== ord('d'):
            print('tecla D pulsada')
            shutil.move(directory+'/'+ filename,directorySaveDescarte+'/'+filename)
            print('movida a Descartes')
    else:
        continue

import os
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.preprocessing import image
from tensorflow.keras import layers, callbacks, optimizers
from tensorflow.keras.models import Sequential, save_model, load_model
import numpy as np
print(tf.__version__)


import time
import datetime
import urllib.request
import socket
socket.setdefaulttimeout(30)

import RPi.GPIO as GPIO
import time
GPIO.setmode(GPIO.BCM)



model = load_model(
    'path to he ".h5" file',
    custom_objects=None,
    compile=True
)

GPIO.setup(21, GPIO.OUT)
for i in range(3): #lets me know that the script is runing
    GPIO.output(21,True)
    time.sleep(0.02)
    GPIO.output(21,False)
    time.sleep(0.02)
GPIO.cleanup()


while True:
  if (int(datetime.datetime.now().strftime("%H"))>5 and int(datetime.datetime.now().strftime("%H"))<17): #todo este jaleo para quedarme con la hora en formato integer del datatime...
  # setting filename and image URL
  try:
    print('dentro del try')
    name=str(datetime.datetime.now())
    filename = '/content/gdrive/MyDrive/Dataset/'+name+'.jpg' #here goes the path to your dataset folder on your google drive
    image_url = "http://212.8.113.121:8081/jpg/1/image.jpg"
    urllib.request.urlretrieve(image_url, filename)
    print('imagen '+name+' descargada.')

    #####
    img = image.load_img(filename, target_size=(150, 150))
    img = np.expand_dims(img, axis=0)
    result_c=model.predict_classes(img)
    print(result_c)
    if result_c==2:
      print('Buenas')
      GPIO.setmode(GPIO.BCM)
      GPIO.setup(21, GPIO.OUT)
      for i in range(3):
        GPIO.output(21,True)
        time.sleep(0.02)
        GPIO.output(21,False)
        time.sleep(0.02)
      GPIO.cleanup()
    elif result_c==1:
      print('Descarte')
    elif result_c==0:
      print('Malas')
    result=model.predict(img)
    print(result)
    print('esperamos...')
    time.sleep(30)
  except:
    print('hay algun problemilla, esperamos medio minuto y reintentamos...')
    time.sleep(30)
  else:
    time sleep(60)