Cyril Diagne's AR Copy Paste App Pulls Objects From Reality and Pastes Them Into Your Laptop
Originally developed as a design prototype, AR Cut and Paste captures real-world objects and pastes them into images — without any editing.
Developer Cyril Diagne has released an augmented reality tool which allows real-world objects to be "cut and pasted" into existing images on a laptop or desktop computer — using nothing more than a smartphone and some machine learning smarts, and written in just one weekend.
"An AR+ML prototype that allows cutting elements from your surroundings and pasting them in an image editing software," Diagne writes of the project. "Although only Photoshop is being handled currently, it may handle different outputs in the future."
In use, the application is impressive: Diagne demonstrates its capabilities by pointing an off-the-shelf smartphone at a range of objects and capturing them automatically before pointing it at the display of a laptop and pasting them — without any distracting background — over an existing image, with no intermediate editing process or even interaction with the laptop itself.
"The secret sauce here is BASNet (Qin et al, CVPR 2019) for salient object detection and background removal," Diagne explains. "The accuracy and range of this model are stunning and there are many nice use cases so I packaged it as a micro-service/docker image. And again, the OpenCV SIFT trick to find where the phone is pointing at the screen. Send a camera image + a screenshot and you get accurate x, y screen coordinates!
"Right now, latency is about ~2.5s for cut and ~4s for paste. There are tons of ways to speed up the whole flow but that weekend just went too fast."
Diagne's prototype is available on GitHub under the permissive MIT licence; following considerable interest he has announced plans to develop it into a finished application dubbed AR Copy Paste, with interested parties able to apply for early access on the dedicated website.
