Researchers Developed an AI that Can ‘Relight’ Portraits After the Fact

julio 16, 2019

Researchers Developed an AI that Can ‘Relight’ Portraits After the Fact

A group of researchers and engineers from UC San Diego and Google have trained a neural network to “relight” portraits after the fact “according to any provided environment map.” In other words: their system can take any photo and adjust the lighting at will—including the direction, temperature, and quality of the light.

The technology was detailed in a research paper submitted to SIGGRAPH 2019z, an annual conference on computer graphics, and like most AI-based image manipulation that we’ve seen lately, the results are impressive to say the least.

The neural network was trained by capturing photos of 18 people under different directional light sources in a studio. Each person was captured from 7 angles at the same time as “a densely sampled sphere of lights” illuminated them from every angle. Based on that data, the researchers were then able to input standard smartphone photos and dynamically “relight” them as if they were captured in a variety of environments:

The researchers demonstrate this using several different photos starting around the 1:55 mark in the video up top. Then, starting around 4:20, they show how the relighting can be applied to the same images while maintaining the background—what they call “illumination retargeting.”

It’s kind of like the Portrait Lighting feature available on iPhones, except this system doesn’t use a depth map or any other data beyond a basic RGB image, and it produces much more dramatic results. What’s more, the paper claims that this technique “can produce a 640 × 640 image in only 160 milliseconds.” That’s not exactly breaking any resolution records, but it gives the researchers hope that “our model may enable compelling consumer-facing photographic relighting applications.”

Speaking of which…

It’s interesting to note that the majority of researchers on this team are part of Google or Google Research, and the lead author, though currently at UC San Diego, was recently a Research Intern at Google himself. When the paper mentions “compelling consumer-facing photographic relighting applications,” we have a feeling those applications might show up for Google Pixel users first.

(via Reddit)