VeLoRA: memory efficient training using Rank-1 sub-token projections
File(s)
Published version
Author(s)
Miles, R
Reddy, P
Elezi, I
Deng, J
Type
Conference Paper
Abstract
Large language models (LLMs) have recently emerged as powerful tools for tackling many language-processing tasks. Despite their success, training and finetuning these models is still far too computationally and memory intensive. In this paper, we identify and characterise the important components needed for effective model convergence using gradient descent. In doing so we find that the intermediate activations used to implement backpropagation can be excessively compressed without incurring any degradation in performance. This result leads us to a cheap and memory-efficient algorithm for both fine-tuning and pre-training LLMs. The proposed algorithm simply divides the tokens up into smaller sub-tokens before projecting them onto a fixed 1-dimensional subspace during the forward pass. These features are then coarsely reconstructed during the backward pass to implement the update rules. We confirm the effectiveness of our algorithm as being complimentary to many state-of-the-art PEFT methods on the VTAB-1k fine-tuning benchmark. Furthermore, we outperform QLoRA for fine-tuning LLaMA and show competitive performance against other memory-efficient pre-training methods on the large-scale C4 dataset. Code: https://github.com/roymiles/VeLoRA.
Date Issued
2025-02-01
Date Acceptance
2024-12-01
Citation
Advances in neural information processing systems, 2025, 37, pp.42292-42310
ISBN
9798331314385
ISSN
1049-5258
Publisher
Neural Information Processing Systems Foundation, Inc. (NeurIPS)
Start Page
42292
End Page
42310
Journal / Book Title
Advances in neural information processing systems
Volume
37
Copyright Statement
© 2025 Neural Information Processing Systems Foundation, Inc. (NeurIPS).
Source
NeurIPS 2024
Publication Status
Published
Start Date
2024-12-10
Finish Date
2024-12-15
Coverage Spatial
Vancouver, Canada