Not in Front of the Radio
RF-Diary provides a textual description of what you are doing behind closed doors.
For reasons including advanced age and illness, some people find themselves in need of assistance to make sure they complete their necessary daily activities, such as taking medications. One solution to this problem is to install an in-home video monitoring system. The usefulness of video monitoring is limited, however, because of an unwillingness by individuals to be monitored in such an invasive way. It is often viewed as an unacceptable invasion of privacy.
The same group at MIT’s Computer Science and Artificial Intelligence Laboratory that recently devised a method to re-identify individuals across time and place using radio frequency (RF) signals has added a new trick to their repertoire that addresses this problem. This time, again using RF to collect data, they have described a new technique called RF-Diary that allows them to provide a textual description of a person’s actions, and interactions with their environment, all while providing a degree of privacy. Only textual descriptions of certain predefined actions need be presented to users of the system.
Dual arrays of 12 antennas, one horizontal and one vertical, transmit frequency-modulated continuous-wave radar signals at an area of interest. The reflected signals are used to generate horizontal and vertical heatmap pairs, at a rate of 30 per second. The heatmaps are fed into a series of ResNet artificial neural networks that generate a skeleton structure of any humans present. The skeleton information is next piped into a hierarchical co-occurrence network, which is a convolutional neural network-based architecture for skeleton action recognition.
Due to peculiarities with RF signals, many objects, especially small objects, are rendered transparent or semi-transparent to the system. To overcome this limitation and still associate the detected actions with objects in the environment, the researchers created a floormap that contains the locations and sizes of objects throughout the house. Given an action, and the proximity to each object, it can then be inferred how a subject was interacting with nearby objects.
To generate the captions, a Long Short Term Memory network was constructed. This network was trained using RF input data generated by volunteers performing various actions. RGB video was also captured alongside the RF to enable Amazon Mechanical Turk workers to label the data.
In testing, RF-Diary was found to generate captions about a person’s activities with greater than 90% accuracy on over 30 different actions. Also, the system managed to generalize to new people and homes it hadn’t previously seen.
Currently, the research team is working to adapt the system to work in real-world homes and hospitals, with the hope of creating a commercial product in the future. There are some great applications for this technology, but there should also be consideration given to possible illicit uses of the device. Peering through walls and darkness to observe people without their knowledge has some very obvious potential for misuse.