The adoption of augmented reality is slowly but surely happening, and it’s easy to see a possible future for technology: hardware that allows you to change what you see in real time, replacing the objects around you with virtual overlays. Call it mixed reality, to be more precise.
Recent research from the Porsche AI team (yes, the automakers – we’ll get to that in a moment) shows how this might work. The team built an AI system dubbed TransforMR that detects objects like cars and people, removes them, and then replaces them with real-time CGI alternatives. The end results are hardly faultless (the changes are random and the CGI models seem to have been borrowed from 3D movie maker) but the concept is striking. It’s not hard to imagine that apps like this will become commonplace in the decades to come.
The team behind the work said The edge that although individual elements of their work have been done before, the composite system is new. TansforMR can work on regular smartphones and tablets, but requires a 4G connection to send data to the cloud. Images are processed in such a way that objects are not just covered up, like with Snapchat AR lenses or Apple’s Memoji, but fully edited. Objects are detected, segmented, then “repainted” (replaced by an AI-generated background) and a CGI model replaces the original.
There are obviously a lot of areas for improvement. Frame rate is only 15 frames per second with low quality paint; the offset is 50 to 100 milliseconds; and CGI replacements are of poor quality. But, the team behind the system says these aspects are relatively easy to improve.
“The main limitation is that large images are very computationally intensive,” said Mohamed Kari, machine learning researcher at Porsche. The edge. “So for inpainting we are currently doing it with very small images, running on 512 x 512 images. But the bandwidth [usage] is negligible. If you can do FaceTime, you can do TransformMR.
One of the key elements of the system, Kari explains, is its use of pose detection. This means that when the system detects a person, for example, it identifies 18 distinct joints in the body. This means that the CGI replacement can be anchored to the movement of the target in real time. Kari compares this to other AR systems that simply identify geometric surfaces.
Watching clips of TransforMR in action, it’s not hard to imagine such software being built into AR glasses. Users could choose a “theme” for their day, replacing cars, buildings and people with sci-fi alternatives or objects from nature. But, as Kari points out, that would involve a huge material challenge. Current augmented reality glasses can only project low-resolution semi-opaque overlays onto their lenses. Right now we just don’t have the technology to “edit” what people see with this type of material. (Although this can presumably be done using a “passthrough” virtual reality system, where first-person cameras broadcast a live video feed to screens that completely obstruct the wearer’s view.)
“We reproduce the full image on the screen, so we can remove whatever we want, but with augmented reality glasses it’s difficult to remove objects because it adds light intensity,” says Kari. “In HoloLens for example, you are looking for through glass, so it is more difficult to remove objects. This question is open to research.
But why is Porsche studying this kind of technology in the first place? According to one of the company’s AI architects, Tobias Grosse-Puppendahl, it’s about improving the experience for passengers and drivers. Future versions of TransforMR software could be used to entertain people stuck in traffic, says Grosse-Puppendahl The edge. “Our main question was: how do we change reality in a fun and entertaining way to react with? And that’s where our idea was born.
Other research projects at Porsche follow a similar theme. For example, the company has also built a prototype system called SoundRide that uses machine vision of a car to detect changes in scenery and select the appropriate music. “Maybe, for example, you’re driving through the Alps, driving on a beautiful road, and all of a sudden you have a great view and maybe the music changes,” says Kari. “We are thinking about how technology can make the in-car experience even more interesting and beautiful.” And that means tinkering with what people would see and hear otherwise.