FitMe: deep photorealistic 3D morphable model avatars
File(s)
Author(s)
Type
Conference Paper
Abstract
In this paper, we introduce FitMe, a facial reflectance model and a differentiable rendering optimization pipeline, that can be used to acquire high-fidelity renderable human avatars from single or multiple images. The model consists of a multi-modal style-based generator, that captures facial appearance in terms of diffuse and specular reflectance, and a PCA-based shape model. We employ a fast differentiable rendering process that can be used in an optimization pipeline, while also achieving photorealistic facial shading. Our optimization process accurately captures both the facial reflectance and shape in high-detail, by exploiting the expressivity of the style-based latent representation and of our shape model. FitMe achieves state-of-the-art reflectance acquisition and identity preservation on single “in-the-wild” facial images, while it produces impressive scan-like results, when given multiple unconstrained facial images pertaining to the same identity. In contrast with recent implicit avatar reconstructions, FitMe requires only one minute and produces relightable mesh and texture-based avatars, that can be used by end-user applications.
Date Issued
2023-08-23
Date Acceptance
2023-06-01
Citation
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp.8629-8640
ISSN
1063-6919
Publisher
IEEE Computer Society
Start Page
8629
End Page
8640
Journal / Book Title
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Copyright Statement
© 2023 IEEE. This CVPR paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore.
Source
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Subjects
Computer Science
Computer Science, Artificial Intelligence
RECONSTRUCTION
Science & Technology
Technology
Publication Status
Published
Start Date
2023-06-17
Finish Date
2023-06-24
Coverage Spatial
Vancouver, Canada