Unlocking the Global Synergies in Low-Rank Adapters

Zhang, Zixi; Zhang, Cheng; Gao, Xitong; Mullins, Robert D; Constantinides, George A; Zhao, Yiren

doi:https://www.dx.doi.org/10.48550/arXiv.2406.14956

File(s)

2406.14956v1.pdf (606.79 KB)

Preprint

OA Location

https://arxiv.org/abs/2406.14956

Author(s)

Zhang, Zixi

Zhang, Cheng

Gao, Xitong

Mullins, Robert D

Constantinides, George A

more

Type

preprint

Abstract

Low-rank Adaption (LoRA) has been the de-facto parameter-efficient fine-tuning technique for large language models. We present HeteroLoRA, a lightweight search algorithm that leverages zerocost proxies to allocate the limited LoRA trainable parameters across the model for better fine-tuned performance. In addition to the allocation for the standard LoRA-adapted models, we also demonstrate the efficacy of HeteroLoRA by performing the allocation in a more challenging search space that includes LoRA modules and LoRAadapted shortcut connections. Experiments show that HeteroLoRA enables improvements in model performance given the same parameter budge. For example, on MRPC, we see an improvement of 1.6% in accuracy with similar training parameter budget. We will open-source our algorithm once the paper is accepted.