Towards resource efficient vision models

Abstract

This PhD thesis focuses on improving the efficiency of deep neural networks for computer vision tasks by employing two key techniques: distillation and pruning. Distillation involves training a smaller network to mimic the behavior of a larger, more complex network, thereby reducing the number of parameters required for accurate inference, reduce the computational complexity, and, in some cases, improve the data-efficiency. Pruning, on the other hand, is typically a post-processing technique that involves removing redundant parameters from the trained network to further reduce its size and computational requirements. This thesis explores various approaches in combining these techniques for enhancing the efficiency of vision models by incorporating domain knowledge and designing novel distillation and pruning techniques. Some of the work presented here also touches upon an orthogonal direction, known as tensor decomposition, which parameterise the weights in a more compact and efficient manner. Overall, this thesis contributes to the development of more efficient and practical deep learning models for computer vision applications. Some examples of these applications may be autonomous driving, surveillance, and augmented virtual reality, but the main emphasis being deploying these models on resource constrained devices, such as mobile phones. The results presented also show various insights into how these efficient models can be designed and trained, thus incorporating all the components of a standard machine learning pipeline, from the architecture design through to the deployment on device.