3D head morphable models and beyond: algorithms and applications

Abstract

It has been more than 20 year since the introduction of 3D morphable models (3DMM) in the computer vision literature. They were proposed as a face representation based on principal components analysis for the task of image analysis, photorealist-manipulation, and 3D reconstruction from single images. Even so, to this date, the applications of such models are limited by a number of factors.

Firstly, training correctly 3DMMs require a vast amount of 3D data that most of the times are not publicly available to the research community due to increasingly stringent data protection regulations. Hence, it is extremely difficult to combine and enrich multiple attributes of the human face/head without the initial 3D images. Additionally, many 3DMMs utilize different templates that describe distinct parts of the human face/head (\ie~face, cranium, ears, eyes) that partly overlap with each other and capture statistical variations which are extremely difficult to incorporate into one single universal morphable model. Moreover, despite the increasing level of detail in the 3D face reconstruction from in-the-wild images, mainly attributed to recent advancements in deep learning, non of the available methods in the literature deal with the human tongue which is important for speech dynamics and improves the realness of the oral cavity. Finally, there is limited work on 3D facial geometric enchantments and translations from different capturing systems due to extremely limited availability of 3D dasasets tailored for this task.

This thesis aims at tackling these shortcomings in all four domains.

A novel approach on how to combine and enrich existing 3DMMs without the underline raw data is proposed. We introduce two methods for solving this problem: i. use a regressor to complete missing parts of one model using the other, ii. use a Gaussian Process framework to blend covariance matrices from multiple models. We show case our approach by combining existing face and head 3DMMs with different templates and statistical variations. Furthermore, we introduce to the research community the first Universal Head Model (UHM) which holds important statistical variation across all key structures of the human head that have an important contribution to to the appearance and identity of a person. We later show case how this model is used to create full head appearances from single in-the-wild images, thus making significant improvements toward the step of realist human head digitization from data-deficient sources.

Additionally, we present the first method that accurately reconstructs the human tongue from single images by utilizing a novel generative framework which models directly the highly deformable surface of the human tongue and seamlessly merges it with our universal head model for more realist representations of the oral cavity dynamics.

Lastly, in this thesis, it is presented a novel generative pipeline capable of converting and enhancing low to high quality 3D facial scans. This will potentially aid depth sensor applications by increasing the quality of the output data while maintaining a low cost. It is also shown that the proposed framework can be extended to handle translations between various expressions on demand.