VidStyleODE: disentangled video editing via StyleGAN and NeuralODEs
Author(s)
Type
Conference Paper
Abstract
We propose VidStyleODE, a spatiotemporally continuous disentangled video representation based upon StyleGAN and Neural-ODEs. Effective traversal of the latent space learned by Generative Adversarial Networks (GANs) has been the basis for recent breakthroughs in image editing. However, the applicability of such advancements to the video domain has been hindered by the difficulty of representing and controlling videos in the latent space of GANs. In particular, videos are composed of content (i.e., appearance) and complex motion components that require a special mechanism to disentangle and control. To achieve this, VidStyleODE encodes the video content in a pre-trained StyleGAN W+ space and benefits from a latent ODE component to summarize the spatiotemporal dynamics of the input video. Our novel continuous video generation process then combines the two to generate high-quality and temporally consistent videos with varying frame rates. We show that our proposed method enables a variety of applications on real videos: text-guided appearance manipulation, motion manipulation, image animation, and video interpolation and extrapolation.
Date Issued
2023-10-02
Date Acceptance
2023-05-01
Citation
Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp.7523-7534
Publisher
Computer Vision Foundation
Start Page
7523
End Page
7534
Journal / Book Title
Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
Copyright Statement
This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. The final published version of the proceedings is available on IEEE Xplore.
Identifier
https://openaccess.thecvf.com/content/ICCV2023/html/Ali_VidStyleODE_Disentangled_Video_Editing_via_StyleGAN_and_NeuralODEs_ICCV_2023_paper.html
Source
IEEE/CVF International Conference on Computer Vision (ICCV)
Publication Status
Published
Start Date
2023-10-02
Finish Date
2023-10-06
Coverage Spatial
Paris, France