Cheering Watch of M5StickC&V

Cheering Watch" to keep an eye on your motions

Getting up in the morning is hard! You live alone, but you need someone to cheer you up! It happens sometimes, doesn't it?

We have created "Cheering Watch," a device that uses the M5StickV's built-in acceleration sensor to estimate your motions and cheer you up according to your motions.

Cheering Watch not only shows you what time it is, but also adds "cheering" to your daily life.

My name is Cheering watch!!

Imaging acceleration data

The acceleration data acquired from the IMU of the M5StickV is converted into an image. This conversion converts the acceleration data into 8x8 RGB format image data, where X-axis = R, Y-axis = G, and Z-axis = B, so that the data advances 8 dots in the X direction and then 8 dots in the Y direction.

When motion is detected by the acceleration sensor, a moiré pattern appears. The moiré pattern depends on the number of vertical and horizontal pixels in the image, the sampling rate of the sensor, and the periodicity of the hand motion.

The data output from the MPU needs to be converted because the minus is a complement.

accel = i2c.readfrom_mem(MPU6886_ADDRESS, MPU6886_ACCEL_XOUT_H, 6)
accel_x = (accel[0]<<8|accel[1])
accel_y = (accel[2]<<8|accel[3])
accel_z = (accel[4]<<8|accel[5])

if accel_x>32768:
    accel_x=accel_x-65536
if accel_y>32768:
    accel_y=accel_y-65536
if accel_z>32768:
    accel_z=accel_z-65536

With M5StickV, save more and more motion data in image data while moving in various ways.

Let's try various ways to move. The resulting image data will be used to train classifications in TensofFlow/Keras.

What you have to be careful about is that the location and orientation of the installation should be fixed when you are learning.

The images of the acceleration data, obtained from the M5StickV and your movements, should be placed in a folder with a different name for each activity.

The more data you take, the better the accuracy of deep learning, so just do lots and lots of different movements! Save image data to SD card.

m5stickv_imu_to_pixel_record.py

Install TensorFlow / Keras for Ubuntu

This session use Ubuntu20.04 or Windows Subsystem for Linux.

Install Miniconda and create a Python environment. The Miniconda installer is downloaded from the Miniconda website.

Miniconda：https://docs.conda.io/en/latest/miniconda.html

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

Install Python, TensorFlow, Keras, etc. on Miniconda.

conda create -n ml python=3.6 tensorflow=1.14 keras pillow \
numpy pydot graphviz

Activate conda.

conda activate ml

Install kendryte nncase

nncase converts learning data created with Keras or TensorFlow into KPU learning data kmodels.

nncase github: https://github.com/kendryte/nncase/

mkdir ./ncc
cd ./ncc
wget https://github.com/kendryte/nncase/releases/download/v0.1.0-rc5/
ncc-linux-x86_64.tar.xz
tar -Jxf ncc-linux-x86_64.tar.xz

Keras learning

A model of acceleration is created by deep learning. Acceleration images are classified into M5StickV SD cards for each activity. Learn with Keras from classification and images at CNN. The learning results are saved with Kmodel.

Run the PYTHON program on Ubuntu as follows command.

python keras_motion_larning.py

keras_motion_larning.py

"my_model.kmodel"file named A is generated.

Kflash GUI

Write m5stickv firmware and kmodel with tools.

Draw Face and Speak

Cheering Watch is equipped with a function that emits a voice in Wav file when you make a predetermined move.

Cheering Watch learns your movements and cheers for you when you want your friends to cheer for you! Cheering Watch will learn your movements and cheer for you when you want it to.

A friendly face is displayed to help you.This face rotates like a gimbal to the tilt. In addition, audio is output according to the motion.

m5stickv_imu_to_pixel_detec.py

Ambient connect

m5stickv cannot be connected to the Internet. So connect it to M5StickC and connect to the Internet. Send motion data to an ambient cloud service in Japan.

The m5stickc software creates programming with arduino.Receive from URT and send to AMBIENT.It also gets the current time from the network and displays the time.

https://github.com/anoken/CheeringWatch_M5StickC-V/blob/master/m5stickc/m5stickc_time_recieve_activity_send_ambient.ino