SyncAI.news, a Varaisys broadcasting
OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloading
JX

Jingyuan Xiao (Tianjin University, Tianjin, China), Jiayue Wang (Tianjin University, Tianjin, China), Yitao Hu (Tianjin University, Tianjin, China), Xinning Wang (Tianjin University, Tianjin, China), Shi Chen (Tianjin University, Tianjin, China), Ziqi Gong (Tianjin University, Tianjin, China), Zhengchao Wang (Tianjin University, Tianjin, China), Guotao Yang (Tianjin University, Tianjin, China), Sheng Chen (Tianjin University, Tianjin, China), Keqiu Li (Tianjin University, Tianjin, China)

· 1 min read

ResearcharXiv cs.CL

OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloading

arXiv:2609.33385v1 Announce Type: cross Abstract: Semi-autoregressive diffusion large language models (dLLMs) improve decoding parallelism through iterative block-wise denoising, but scaling them with mixture-of-experts (MoE) layers introduces a large expert parameter footprint that exceeds memory-constrained GPU capacity. Expert offloading is a natural remedy, yet existing MoE serving systems target autoregressive decoding and rely on intra-iteration layer-wise prefetching: while computing one layer, they predict and load experts for subsequent layers. Under dLLM inference, block-wise routing expands the active expert working set within each iteration, making such prefetches difficult to complete in time and costly when mispredicted. Consequently, existing prefetch-based solutions often degenerate into on-demand expert loading with high decoding latency. We propose OLED-MoE, an expert offloading system that shifts the optimization target from intra-iteration prefetching to inter-iteration expert retention. Its key insight is that adjacent denoising iterations exhibit strong expert routing overlap, and token confidence indicates which experts are likely to be reused. OLED-MoE uses confidence-guided inter-iteration prediction to retain high-value experts in GPU memory without introducing extra prefetch traffic. It further compensates unavoidable cache misses through CPU-GPU cooperative execution, jointly considering dynamic expert computation load and predicted future reuse. Across diverse dLLM workloads, OLED-MoE reduces time per output token (TPOT) by 1.23x-7.93x and improves expert cache utilization by 1.44x-4.23x over state-of-the-art offloading systems. Notably, OLED-MoE approaches full-residency performance while using only 40% of the expert GPU memory, incurring merely 23% higher TPOT despite a 60% reduction in expert memory footprint. OLED-MoE's source code is publicly available at https://github.com/flashserve/OLED-MoE.

Original source

This story was published by arXiv cs.CL and written by Jingyuan Xiao (Tianjin University, Tianjin, China), Jiayue Wang (Tianjin University, Tianjin, China), Yitao Hu (Tianjin University, Tianjin, China), Xinning Wang (Tianjin University, Tianjin, China), Shi Chen (Tianjin University, Tianjin, China), Ziqi Gong (Tianjin University, Tianjin, China), Zhengchao Wang (Tianjin University, Tianjin, China), Guotao Yang (Tianjin University, Tianjin, China), Sheng Chen (Tianjin University, Tianjin, China), Keqiu Li (Tianjin University, Tianjin, China). SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News