
ZX
Zebang Xie, Chuanyang Zheng, Yik-Chung Wu, Yihang Gao
· 1 min read
ResearcharXiv cs.LG
Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization
arXiv:2609.28998v1 Announce Type: new
Abstract: Low-rank adaptation (LoRA) has become a popular parameter-efficient fine-tuning method for large language models. A key challenge in LoRA is how to determine the rank of each adaptation matrix, as rank directly controls its capacity and efficiency. Existing adaptive-rank methods typically allocate ranks according to manually designed importance scores, which are not directly derived from an optimization objective. In this work, we propose $\ell_p$-LoRA, a principled rank-allocation method based on $\ell_p$ regularization with $0
Original source
This story was published by arXiv cs.LG and written by Zebang Xie, Chuanyang Zheng, Yik-Chung Wu, Yihang Gao. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on arxiv.org


