SyncAI.news, a Varaisys broadcasting
Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization
ZX

Zebang Xie, Chuanyang Zheng, Yik-Chung Wu, Yihang Gao

· 1 min read

ResearcharXiv cs.LG

Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via lp Regularization

arXiv:2609.28998v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) has become a popular parameter-efficient fine-tuning method for large language models. A key challenge in LoRA is how to determine the rank of each adaptation matrix, as rank directly controls its capacity and efficiency. Existing adaptive-rank methods typically allocate ranks according to manually designed importance scores, which are not directly derived from an optimization objective. In this work, we propose $\ell_p$-LoRA, a principled rank-allocation method based on $\ell_p$ regularization with $0

Original source

This story was published by arXiv cs.LG and written by Zebang Xie, Chuanyang Zheng, Yik-Chung Wu, Yihang Gao. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News