VkD: Improving knowledge distillation using orthogonal projections
Author(s)
Miles, Roy
Elezi, Ismail
Deng, Jiankang
Type
Conference Paper
Abstract
Knowledge distillation is an effective method for training small and efficient deep learning models. However, the efficacy of a single method can degenerate when transferring to other tasks, modalities, or even other architectures. To address this limitation, we propose a novel constrained feature distillation method. This method is derived from a small set of core principles, which results in two emerging components: an orthogonal projection and a task-specific normalisation. Equipped with both of these components, our transformer models can outperform all previous methods on ImageNet and reach up to a 4.4% relative improvement over the previous state-of-the-art methods. To further demonstrate the generality of our method, we apply it to object detection and image generation, whereby we obtain consistent and substantial performance improvements over state-of-the-art. Code and models are publicly available11https://github.com/roymiles/vkd.
Date Issued
2024-09-16
Date Acceptance
2024-06-01
Citation
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp.15720-15730
ISSN
1063-6919
Publisher
IEEE
Start Page
15720
End Page
15730
Journal / Book Title
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Copyright Statement
© 2024 IEEE. This CVPR paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore.
Source
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publication Status
Published
Start Date
2024-06-16
Finish Date
2024-06-22
Coverage Spatial
Seattle, WA, USA