Apple with Neural 3D Engine

With FaceLit, Apple proposes a neural 3D engine for face relighting. The engine should

Generate 3D faces that can be rendered in various custom lighting conditions and views. The engine only needs 2D images as input. The engine relies on standard pose and lighting estimators. With these estimates, the Phong reflection model is integrated into the neural volume rendering system. The model learns how to generate the shape and material properties of a face in this way. This is also the basis for generating photorealistic facial images with multiview 3D and lighting consistency.

Apple confidently states: "Our method enables photorealistic generation of faces with explicit lighting and view controls on multiple datasets - FFHQ, MetFaces and CelebA-HQ. We demonstrate the most modern photorealism among 3D-aware GANs on the FFHQ dataset and achieve a FID -Score of 3.5."

With Relight FX, Blackmagic has already integrated a similar function into the new DaVinci Resolve 18.5 BETA. This AI effect makes it possible to subsequently illuminate scenes with virtual light sources in Resolve. The key difference: this effect works with full videos and does not produce new perspective views.