PeopleLens Tasks AI to Support Social Interaction for Blind Kids
Microsoft's AI platform helps people who are blind or who have low vision make sense of and engage with their immediate social surroundings.
Children who are visually impaired often have difficulty participating in social engagements. Unsure where to direct their voices, especially in groups, or the inability to pick up on body language can be a struggle when trying to befriend other kids in the same age groups. The same can be said for support groups, families, and teachers who can find it equally difficult to help those kids form those bonds and connections.
To help overcome those challenges, Microsoft Research has developed the PeopleLens platform – an open-lens, AI-driven system that helps kids stay aware of the locations and identities of those around them. The system is designed around Nreal Light AR glasses and connected to a smartphone outfitted with computer vision algorithms and a clever set of tools that draw on studies and experience from psychology and speech and language therapy. The device reads aloud the names of individuals the wearer is facing using spatial audio, meaning the sound comes from the person's direction, enabling the wearer to understand position and distance.
Those computer vision algorithms allow the headset to continuously locate, identify, track and capture the gaze direction of those in the immediate area of the wearer via audio in real-time, providing a level of immersion and inclusion in social situations. Cameras equipped on the headwear receive images that are also processed by those CV algorithms to provide a series of overlays that are stitched together to create a map of those in the vicinity. This helps keep track of the names and locations the wearer is interacting with at any given time. Of course, no video or identifying information is saved, so the system can't be used for malicious intent.
The system works by employing a series of sounds to assist the wearer in placing people in the surrounding space. For example, a percussive bump indicates a gaze has crossed a person up to 10-meters away. The bump is followed by the person's name (if registered in the system), is within four meters, and both their ears can be detected. The sound of woodblocks guides the wearer in finding and centering on the person's face, which the system has "seen" for one second but hasn't identified, changing the pitch of the woodblocks accordingly. A gaze notification can alert the wearer when they are being looked at, letting the wearer initiate interaction.
The PeopleLens is still in the prototype phase and undergoing development with the help of the kids and people the system will eventually serve. Microsoft is currently looking for others interested in participating in that endeavor to help further evolve the technology.