FitDiff: Robust monocular 3D facial shape and reflectance estimation using Diffusion Models
Author(s)
Galanakis, Stathis
Lattas, Alexandros
Moschoglou, Stylianos
Zafeiriou, Stefanos
Type
Conference Paper
Abstract
The remarkable progress in 3D face reconstruction has resulted in high-detail and photorealistic facial representations. Recently, Diffusion Models have revolutionized the capabilities of generative methods by surpassing the performance of GANs. In this work, we present FitDiff, a diffusion-based 3D facial avatar generative model. Lever-aging diffusion principles, our model accurately generates relightable facial avatars, utilizing an identity embedding extracted from an “in-the-wild” 2D facial image. The introduced multimodal diffusion model is the first to concurrently output facial reflectance maps (diffuse and specular albedo and normals) and shapes, showcasing great generalization capabilities. It is solely trained on an annotated subset of a public facial dataset, paired with 3D reconstructions. We revisit the typical 3D facial fitting approach by guiding a reverse diffusion process using perceptual and face recognition losses. Being the first 3D LDM conditioned on face recognition embeddings, FitDiff reconstructs relightable human avatars, that can be used as-is in common rendering engines, starting only from an unconstrained facial image, and achieving state-of-the-art performance.
Date Issued
2025-04-08
Date Acceptance
2025-02-01
Citation
2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025, pp.992-1004
ISBN
979-8-3315-1084-8
ISSN
2472-6737
Publisher
IEEE Computer Society
Start Page
992
End Page
1004
Journal / Book Title
2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
Copyright Statement
Copyright © 2025, IEEE. This is the author’s accepted manuscript made available under a CC-BY licence in accordance with Imperial’s Research Publications Open Access policy (www.imperial.ac.uk/oa-policy)
License URL
Source
2025 Winter Conference on Applications of Computer Vision-WACV
Subjects
Computer Science
Computer Science, Artificial Intelligence
Computer Science, Theory & Methods
Imaging Science & Photographic Technology
Science & Technology
Technology
Publication Status
Published
Start Date
2025-02-28
Finish Date
2025-03-04
Coverage Spatial
Tucson, AZ, USA