Controllable person image synthesis with pose-constrained latent diffusion
Author(s)
Han, Xiao
Zhu, Xiatian
Deng, Jiankang
Song, Yi-Zhe
Xiang, Tao
Type
Conference Paper
Abstract
Controllable person image synthesis aims at rendering a source image based on user-specified changes in body pose or appearance. Prior art approaches leverage pixel-level denoising diffusion models conditioned on the coarse skeleton via cross-attention. This leads to two limitations: low efficiency and inaccurate condition information. To address both issues, a novel Pose-Constrained Latent Diffusion model (PoCoLD) is introduced. Rather than using the skeleton as a sparse pose representation, we exploit DensePose which offers much richer body structure information. To effectively capitalize DensePose at a low cost, we propose an efficient pose-constrained attention module that is capable of modeling the complex interplay between appearance and pose. Extensive experiments show that our PoCoLD outperforms the state-of-the-art competitors in image synthesis fidelity. Critically, it runs 2× faster and consumes 3.6× smaller memory than the latest diffusion-model-based alternative during inference. Our code and models are available at https://github.com/BrandonHanx/PoCoLD.
Date Issued
2024-01-15
Date Acceptance
2023-10-01
Citation
2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2024, pp.22711-22720
ISSN
1550-5499
Publisher
IEEE Computer Society
Start Page
22711
End Page
22720
Journal / Book Title
2023 IEEE/CVF International Conference on Computer Vision (ICCV)
Copyright Statement
© 2023 IEEE. This ICCV paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore.
Source
IEEE/CVF International Conference on Computer Vision (ICCV)
Subjects
Computer Science
Computer Science, Artificial Intelligence
Computer Science, Theory & Methods
Imaging Science & Photographic Technology
Science & Technology
Technology
Publication Status
Published
Start Date
2023-10-02
Finish Date
2023-10-06
Coverage Spatial
Paris, France