Emotional Assistant Robot

Introduction

Nowadays, it is not uncommon to find intelligent household appliances, such as washing machines or refrigerators, indispensable in everyday life. This is something that also aspires to welfare robotics, which develops automatons specializing in caring for people with some dependency, such as robots nurse who remind patients to take medicines. Their aim is to integrate these robots into the home as if they were just another appliance, so that it is not uncommon to live with them. This type of robots have complex algorithms that allow them to interact and identify people or recommend activities. However, the vast majority are large and expensive.

With the latest advances in IoT devices, with greater computing capacity and the incorporation of small neural networks, their applications in this area can become of great interest. Moreover, in recent years, robotics has been simplified so that anyone can have access to uncomplicated levels. In fact, it is common for people with little knowledge of programming and electronics to design and build their own robot, all thanks to the rise of open source prototype platforms.

In this sense, the following project is presented, whose objective was the creation of a low-cost assistant robot that incorporated an IoT device. To carry out this project, the Sony Spresense development system was used (Figure 1).

This new card incorporates a multicore (6 core) CXD5602 processor, camera, a high-resolution audio system and multi-micro inputs - 192kHz/24-bit advanced audio codec and amplifier for audio output, and support for up to 8 microphone input channels, integrated GNSS with support for GPS, QZSS and GLONASS allowing applications where tracking is required. These features make this development system a useful and ideal tool for applications in robotics, IoT, tracking, etc...

Project Description

According to the World Health Organization WHO, it is estimated that by 2050 people over 60 years of age will double. This creates a problem related to the logistics involved in assisting these people, so that the demand for trained personnel to assist older people will increase. In order to meet this need, robotics represents a good option, being in the market robots that remind people of the schedule of their medicines. These robots are generally station robots, that is to say that they are fixed in a specific place of our homes.

The robot presented in this project is a first approximation of an accompanying robot, which can be used as a station robot or as a robot that accompanies the user everywhere. Some of these robots have been observed, one of the first robots of this type was the Archimedes robot.

(https://www.hackster.io/glowascii/archimedes-the-ai-robot-owl-325ff5), the next one was the robot Asi.

Robot Construction

The construction of the robot is divided into three phases. The first is the mechanical design of the robot using 3D design software (SolidWorsk). The second phase focused on low level programming, i.e. Spresense programming (camera control, access to WiFi, control of motor servos. Finally, the last phase focused on the programming of the web service based on flask, in charge of carrying out the analysis of the information obtained by the robot.

Design and construction of the robot.

To design the robot a SolidWorks software was used to make the 3D modeling that would be printed later. The robot has two parts, a head which houses the electronics and a base that allows you to move the head.

Figure 2 shows the printed robot. The robot has two eyes and a mouth, in which the camera has been placed (Figure 3).

The Spresense development system does not have a WiFi module, so it was decided to add an ESP-8266. This module allows the robot to send the information to the web service, Figure 4 shows the location of the ESP-8266 inside the head.

In order to accommodate the electronics inside the head, a piece was designed to hold the Spresense (Figure 5).

To perform the movements of the robot was incorporated ajoint in the neck, which allows the head to move from left to right, forwardand back (Figure 6).

Spresense Programming

The Spresense is a development systemthat has a multicore processor, which incorporates different elements such ascamera, high definition audio and GPS. However, it does not have integratedcommunication modules (WiFi or Bluetooth) that allow communication with the outside.For this reason, an ESP-8266 has been added, with which the system can send theacquired data to the web service. Within the information to send we find theimage acquired by the camera, GPS and other sensors.

One of the inconveniences that arose atthe time of making this communication, was in the sending of the image. Thecamera captures images of 640x480, so we saw the need to resize the image. Thisoption is not integrated into the Spresense camera library, so we opted to usethe JPEGDecoder library, which was modified to work with the Spresense system.It is important to note that the image has to be sent in grayscale and not inRGB, and must be sent in packages that do not exceed 1024 bytes, which is the maximumsize of the data sending buffer of ESP-8266.

These images are sent to the web servicedeveloped in Flask. This web service is in charge of reconstructing the imagesent by the robot. To reconstruct the image it is necessary that the robotsends in the first bits the size of the image and then the information of eachone of the pixels and an "END" to indicate that the sending wasfinished.

Web Service

Once the reconstruction is done, the system analyzes the image,identifying objects or detecting faces. To make these identifications thesystem uses different artificial intelligence algorithms, which were trainedfor public databases such as Ciraf-100(https://www.cs.toronto.edu/~kriz/cifar.html). At the same time we created adatabase of objects that we can find in our home, such as spoons, lamps,tables, etc.. All these images were pre-processed and resized to 32x32 pixelsto facilitate the training of the different neural networks. At the same timethe system allows the identification of people and the classification ofemotional states. To perform this task the web service needs to locate the facewithin the image. Using the "dlib" library for Python, it is possibleto detect the face and extract the main characteristics for the identificationof the person. In order to carry out the identification process, it isnecessary to encode the image in a vector. This vector is a one-dimensionalarray in which the image has been coded and has a size of 1x128. This codingallows the robot to recognize the person only with an image, if compared withother techniques (neural networks and deep learning) this technique is fasterand does not require lengthy learning processes. Figure 7 and 8 show the vectorfor two users and Figure 9 shows the comparison between them. Foridentification purposes, the system calculates the distance between the twovectors and if this distance is within a threshold, it means that it is theperson.

The classification of emotions was carried outusing the KDEF data set (http://www.kdef.se). Seven emotional states areclassified, AF = afraid, , AN = angry, , DI = disgusted, , HA = happy, NE =neutral, SA = sad, SU = surprised. To classify these emotions the deep-learningtool was used using tensor flow as a development tool. This database has a total of 4900 images witha size of 562 * 762 pixels and a resolution of 72*72 dpi. The dataset wasdivided into 80% for training, 10% for testing and another 10% forvalidation. The results obtainedfrom the learning process can be seen in Figures 10 and 11.

Video: