AI-Backed Wearable Cameras Can Save Patients From Drug Mix-Ups, Researchers Say

Researchers from the University of Washington, Carnegie Mellon University, Makerere University, and the Toyota Research Institute in Los Altos are looking to make medication mix-ups a thing of the past — by using wearable cameras and a computer vision system to double-check patients are getting what they should be getting.

"The thought of being able to help patients in real time or to prevent a medication error before it happens is very powerful," says co-lead author Kelly Michaelsen, assistant professor of anesthesiology and pain medicine, of the team's work. "One can hope for a 100 percent performance but even humans cannot achieve that. In a survey of more than 100 anesthesia providers, the majority desired the system to be more than 95 percent accurate, which is a goal we achieved.”

The idea: reducing the one-in-twenty patients who are affected by preventable harm in a clinical setting, with a focus on those harmed by mistakes made during medication — incidents that, in up to 12 percent of cases, lead to serious harm or death, either through incorrect dosage or dispensing the wrong medicine. To do this, the team aims to give staff an artificial intelligence (AI) assistant — monitoring what's happening through a head-mounted GoPro camera and alerting to potential errors in real-time.

"It was particularly challenging, because the person in the OR [Operating Room] is holding a syringe and a vial, and you don't see either of those objects completely," co-author Shyam Gollakota, a professor at the Paul G. Allen School of Computer Science and Engineering at the University of Washington. "Some letters (on the syringe and vial) are covered by the hands. And the hands are moving fast. They are doing the job. They aren't posing for the camera."

The deep learning model developed by the team, which was trained on a custom dataset comprised of high-resolution videos drug-draws carried out by 13 anesthesiology providers across a number of different operating rooms, doesn't just look for barcodes or attempt to recognize the text on each vial, the team says: it looks for visual clues including vial size, syringe size, cap color, and even the size of the print on the label. It also focuses exclusively on the drugs in-hand, ignoring any others in the room.

In testing, the system showed promise: 99.6 percent sensitivity and 98.8 specificity at detecting vial-swap errors. "AI is doing all that," Gollakota of the test results. "Detecting the specific syringe that the healthcare provider is picking up, and not detecting a syringe that is lying on the table."

The team's work has been published in the journal NPJ Digital Medicine under open-access terms.