Monocular identity-conditioned facial reflectance reconstruction
Author(s)
Type
Conference Paper
Abstract
Recent 3D face reconstruction methods have made re-markable advancements, yet there remain huge challenges in monocular high-quality facial reflectance reconstruction. Existing methods rely on a large amount of light-stage captured data to learn facial reflectance models. However, the lack of subject diversity poses challenges in achieving good generalization and widespread applicability. In this paper, we learn the reflectance prior in image space rather than UV space and present a framework named ID2Reflectance. Our framework can directly estimate the reflectance maps of a single image while using limited reflectance data for training. Our key insight is that reflectance data shares facial structures with RGB faces, which enables obtaining expressive facial prior from inexpensive RGB data thus re-ducing the dependency on reflectance data. We first learn a high-quality prior for facial reflectance. Specifically, we pretrain multi-domain facial feature code books and design a codebook fusion method to align the reflectance and RGB domains. Then, we propose an identity-conditioned swapping module that injects facial identity from the target image into the pre-trained autoencoder to modify the identity of the source reflectance image. Finally, we stitch multi-view swapped reflectance images to obtain renderable assets. Extensive experiments demonstrate that our method exhibits excellent generalization capability and achieves state-of-the-art facial reflectance reconstruction results for in-the-wild faces. Our project page is https://xingyuren.github.io/id2reflectance.
Date Issued
2024-09-16
Date Acceptance
2024-06-01
Citation
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp.885-895
ISSN
1063-6919
Publisher
IEEE
Start Page
885
End Page
895
Journal / Book Title
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Copyright Statement
© 2024 IEEE. This CVPR paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore.
Source
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publication Status
Published
Start Date
2024-06-16
Finish Date
2024-06-22
Coverage Spatial
Seattle, WA, USA