Advances of graph neural networks for 3D shape learning and analysis

Abstract

In recent years, there has been a surge in the availability of 3D sensors, leading to an exponential increase in the amount of 3D data, paving the way for remarkable advancements in 3D computer vision applications. The advances of deep learning on geometric data with irregular structures, such as meshes and point clouds, have further enhanced the ability to analyze and understand 3D shapes.

In this thesis, we explore the usage of geometric deep learning methods in 3D shape analysis. The thesis can be divided into two parts. In the first part, the advancements of non-linear localized mesh convolutions to tackle the limitations of traditional statistical shape modeling methods, such as PCA, to capture fine details and extreme deformations given its linear structure and global structure is shown. In particular, the cases of static and dynamic neural deformable models are explored using a localized mesh convolution operator to generate high fidelity deformable models. Using a large scale dataset of hand scans composed by over 1200 subjects, a fine-grained neural hand model is constructed that is able to outperform current state-of-the-art hand models. To explore the expressive power of graph convolution in dynamic morphable models, a 4D generative model is proposed that is able to manipulate 3D faces and generate dynamic expressions fully customized by the user. Both models achieve state-of-the-art performance that outperforms traditional PCA deformable models.

However, dealing with high-fidelity models poses several challenges, especially when it comes to processing and storage. The enormous amount of points needed to capture the fine-grained features of such models can be computationally expensive and memory-intensive which limits their real-time applications. In the second part of this thesis, two neural based simplification methods are proposed to simplify point clouds and meshes in real-time. Both methods rely on graph neural networks to capture rich local and global topological information of the 3D objects. Initially, a point cloud simplification method is proposed that samples points in an sophisticated matter to preserve the underlying perceptual features of the point cloud. Then, the simplification method is extended to meshes through a graph neural network triangulation module that constructs the faces of the simplified mesh. Through extensive evaluations and comparisons with state-of-the-art baselines, we demonstrate the effectiveness and efficiency of our method in preserving important shape characteristics while significantly reducing the data size.