SyncAI.news, a Varaisys broadcasting
Localized LoRA-MoE: Mixture of Low-Rank Experts With Adaptive Routing
BB

Babak Barazandeh, Subhabrata Majumdar, Vinay Prithyani, George Michailidis

· 1 min read

ResearcharXiv cs.CL

Localized LoRA-MoE: Mixture of Low-Rank Experts With Adaptive Routing

arXiv:2607.05114v2 Announce Type: replace-cross Abstract: Parameter-efficient fine-tuning (PEFT) methods such as LoRA adapt large models through a single low-rank update. Such updates are susceptible to gradient interference: when a training stream interleaves multiple operational contexts, conflicting updates to shared adapter weights collapse the adapter into an unspecialized average. While recent methods introduce block-wise isolation to remedy this issue, their topologies are static and cannot adapt to context switching or localized sensor degradation. In this work, we introduce Localized LoRA-MoE, a framework that combines localized spatial blocking with context-conditioned expert routing. We propose and evaluate two architectural variants: Block-Wise LoRA-MoE (centralized routing), which modulates the entire structural grid through a single context-conditioned gate, and Cell-Wise LoRA-MoE (decentralized routing), which assigns an independent gate to every coordinate cell of the matrix grid. Across three structured transformation benchmarks--a high-dimensional SVD matrix simulation, context-switched transformations over real tabular data, and spatial reconstruction under simulated sensor occlusion--both variants outperform static baselines under comparable adapter budgets. Both variants perform similarly when context shifts the grid uniformly, while cell-wise routing is strictly better when contexts are spatially heterogeneous. When fine-tuning a language model on six interleaved GLUE tasks, task-conditioned routing at a fixed adapter budget outperforms LoRA and static localized adapters, matching per-task adapters with 6x budgets. At decoder scale on Qwen2.5-1.5B, both routed variants outperform static baselines across eight commonsense benchmarks, with cell-wise routing achieving +3.9 points over LoRA at matched active capacity.

Original source

This story was published by arXiv cs.CL and written by Babak Barazandeh, Subhabrata Majumdar, Vinay Prithyani, George Michailidis. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News