
QS
Qi Sun, Siyue Zhang, Tingyu Song, Yilun Zhao, Chen Zhao
· 1 min read
ResearcharXiv cs.LG
Rethinking Offline On-Policy Distillation: From Scalar Advantages to Distributional Supervision
arXiv:2611.18648v1 Announce Type: new
Abstract: On-policy distillation (OPD) transfers teacher capabilities through token-level feedback on student-generated trajectories, but repeatedly evaluating fresh rollouts with the teacher is computationally expensive. Offline methods such as Lightning OPD reduce this cost by caching recorded tokens' teacher log-probabilities on fixed reference rollouts. By extending the idea to a multi-turn agentic search setup, we find that it consistently underperforms the SFT starting point trained by the same teacher. We empirically observe greater teacher--reference mismatch than in mathematical reasoning and post-training distributional flattening, and theoretically show that scalar advantage supervision on fixed rollouts admits solutions with flattened output distributions under suitable conditions. Motivated by these findings, we propose Distributional On-Policy Distillation (DistOPD), a simple yet effective offline OPD method that retains candidate-token probabilities from both reference and teacher at selected positions and aggregates the remaining parts into a residual token. On agentic search, DistOPD achieves performance comparable to online OPD while improving student--teacher alignment and using training GPU-hours comparable to Lightning OPD.
Original source
This story was published by arXiv cs.LG and written by Qi Sun, Siyue Zhang, Tingyu Song, Yilun Zhao, Chen Zhao. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on arxiv.org


