Automatic detection of activities in first person videos

Identify activities of daily life, through the detection of objects in videos taken with a first-person camera and assist with voice.

AdvancedFull instructions provided24 hours117

Automatic detection of activities in first person videos

Things used in this project

Hardware components

NVIDIA Jetson Nano Developer Kit

Google AIY Voice

Raspberry Pi Zero

Software apps and online services

Raspberry Pi Raspbian

Linux Driver Package

Google DialogFlow

Hand tools and fabrication machines

3D Printer (generic)

Story

Occupational therapy professionals are capable of guiding people with cognitive deficits to acquire necessary skills in their daily lives, to do so they focus on functional cognition that includes addressing the relationship between people, their daily occupations, and the context. Cognitive rehabilitation is dedicated to restoring mental processes such as memory, attention, understanding, problem-solving, reasoning, self-awareness, and perception, using designed techniques and teaching compensatory strategies through cognitive interventions that focus on cognitive functioning rather than behavioral functioning, Cognitive rehabilitation allows us to compensate or prevent the loss of cognitive functioning and promote people's well-being. Cognitive interventions are administered in all types of populations; however, for the intervention to be successful, the group or individual characteristics of the people with the injury or illness must be considered. People with alterations in cognitive functions must be constantly monitored and for this, the use of artificial intelligence shows good performance in detection and monitoring applications of people in real-time. On the other hand, compensation is aimed at methods that overcome or solve difficulties that arise when carrying out a task, function, or process, allowing them to be completed successfully and adapted to the patient. Hence, training in specific tasks such as activities of daily living, where cognitive rehabilitation aims to improve the performance of functional tasks through goal-directed practice and repetition. When analyzing the activities of daily living, it is essential to consider the tasks that compose them and the functional challenges of each task. The following figure shows the breakdown of some of these activities.

This proposal proposes the construction of a support system for activities of daily living in patients who cannot feed themselves. Finally, recognizing daily life activities by voice is a simple task since a simple question is enough to identify places and actions carried out, instead of assisting. step by step of an activity is complex since it is easy to deviate from the established flow, this is why it is necessary to work hand in hand with health professionals to characterize the flow of conversation according to the disability, the age of the user, or any other variable. that could affect the flow. Hence, our development has a voice assistant that can be activated or deactivated when the person requests help and not at all times.

Code

transfer_learning_license_kitchen_recognition.py

# -*- coding: utf-8 -*-
"""Transfer Learning license kitchen recognition.ipynb

Automatically generated by Colaboratory.

Original file is located at
    https://colab.research.google.com/drive/1XoPOF60tM-7hot61yhOnXEiUVA-ZaQjS
"""

!nvidia-smi

!pip install onnx

from google.colab import drive
drive.mount('/gdrive')
# Si colocan algun archivo nuevo
#drive.mount("/gdrive", force_remount=True)

import os
os.chdir('/gdrive/MyDrive/UR/JetsonNano/ssd')
!ls

#!unzip '/content/drive/MyDrive/colabs/JetsonNano/ssd.zip'

os.chdir('/gdrive/MyDrive/UR/JetsonNano/ssd')
!ls

!wget https://nvidia.box.com/shared/static/djf5w54rjvpqocsiztzaandq1m3avr7c.pth -O models/mobilenet-v1-ssd-mp-0_675.pth

!pip install boto3

!python open_images_downloader.py --class-names 'Refrigerator,Gas stove,Mug,Kitchen & dining room table,Blender,Cabinetry,Plate,Wok,Kitchen utensil,Bowl' --data=data/plate

!python train_ssd.py --data=data/plate --model-dir=models/plate --batch-size=128 --num-epochs=5 --resume models/plate/mb1-ssd-Epoch-4-Loss-3.3897489309310913.pth

!python onnx_export.py --input=models/plate/mb1-ssd-Epoch-4-Loss-3.3897489309310913.pth --output=./models/ssd_kitchen_2.onnx --labels=./models/plate/labels.txt

!pip install jetson-emulator

import jetson_emulator

import jetson_emulator.inference as inference
import jetson_emulator.utils as utils

# load the recognition network
net = inference.imageNet("googlenet")
for x in range(1,6):
	# emulator API to generate sample images for imageNet
	filename = net.emulatorGetImageFile()
	img = utils.loadImage(filename)
	class_idx, confidence = net.Classify(img)
	class_desc = net.GetClassDesc(class_idx)
	print("image "+str(x)+" is recognized as '{:s}' (class #{:d}) with {:f}% confidence".
	format(class_desc, class_idx, confidence*100))

Credits

Oscar Julian Perdomo Charry

2 projects • 2 followers

Associate professor, Faculty of Engineering - Department of Electric and Electronic Engineering, Universidad Nacional de Colombia

HERNAN BERNAL

0 projects • 1 follower

Thanks to Elkin Estiven González Cuellar and Rolffer Alexander Ramírez Serrano.

Automatic detection of activities in first person videos