SVD-NAS: coupling low-rank approximation and neural architecture search
File(s)
Accepted version
Author(s)
Yu, Zhewen
Bouganis, Christos-Savvas
Type
Conference Paper
Abstract
The task of compressing pre-trained Deep Neural Networks has attracted wide interest of the research community due to its great benefits in freeing practitioners from data access requirements. In this domain, low-rank approximation is a promising method, but existing solutions considered a restricted number of design choices and failed to efficiently explore the design space, which lead to severe accuracy degradation and limited compression ratio achieved. To address the above limitations, this work proposes the SVD-NAS framework that couples the domains of low-rank approximation and neural architecture search. SVD-NAS generalises and expands the design choices of previous works by introducing the Low-Rank architecture space, LR-space, which is a more fine-grained design space of low-rank approximation. Afterwards, this work proposes a gradient-descent-based search for efficiently traversing the LR-space. This finer and more thorough exploration of the possible design choices results in improved accuracy as well as reduction in parameters, FLOPS, and latency of a CNN model. Results demonstrate that the SVD-NAS achieves 2.06-12.85pp higher accuracy on ImageNet than state-of-the-art methods under the data-limited problem setting. SVD-NAS is open-sourced at https://github.com/Yu-Zhewen/SVD-NAS.
Date Issued
2023-02-06
Date Acceptance
2023-01-02
Citation
2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023, pp.1503-1512
ISBN
978-1-6654-9346-8
ISSN
2472-6737
Publisher
IEEE
Start Page
1503
End Page
1512
Journal / Book Title
2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
Copyright Statement
Copyright © 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Identifier
https://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=PARTNER_APP&SrcAuth=LinksAMR&KeyUT=WOS:000971500201058&DestLinkType=FullRecord&DestApp=ALL_WOS&UsrCustomerID=a2bf6146997ec60c407a63945d4e92bb
Source
23rd IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
Subjects
Computer Science
Computer Science, Artificial Intelligence
Engineering
Engineering, Electrical & Electronic
Imaging Science & Photographic Technology
Science & Technology
Technology
Publication Status
Published
Start Date
2023-01-02
Finish Date
2023-01-07
Coverage Spatial
Waikoloa, HI, USA
Date Publish Online
2023-02-06