Deep learning is now a predominant technique for most machine learning problems, especially in computer vision. With multiple layers, deep networks can extract hierarchical features from the images to solve highly non-linear problems. Currently, its development has been taken into many directions, such as modifying networks' architectures, using different losses during training, and adjusting the training strategies. Regarding facial analysis in imagery, it has been advantageous to learn statistical models for both appearances and shapes, which has not been fully explored in the context of deep learning. In this thesis, we examine how deep learning can be carefully designed to solve face analysis in both discriminative and generative settings more effectively. In particular, we incorporated the impressive feature extraction power from deep networks with spatial features provided by well-studied facial landmarks. Our proposed methods further improved the performance in each task, suggesting that deep learning can benefit from utilizing statistical prior associated with each problem.