
JX
Jing Xiong, Hui Shen, Shansan Gong, Yuxin Cheng, Jianghan Shen, Chaofan Tao, Haochen Tan, Haoli Bai, Lifeng Shang, Ngai Wong
· 1 min read
ResearcharXiv cs.CL
OVD: On-policy Verbal Distillation
arXiv:2601.21968v2 Announce Type: replace
Abstract: Knowledge distillation transfers reasoning capabilities from large teachers to efficient students. However, token-level on-policy
distillation (OPD) constrains student exploration and requires teacher token probabilities, precluding distillation from black-box teachers
that provide only text outputs. We introduce On-policy Verbal Distillation (OVD), a framework that uses verbal scores from black-box
teachers to rank student-generated sub-trajectories, retaining high-scoring ones and replacing low-scoring ones with teacher-generated
continuations. We analyze when ranking induced by verbal scores can guide distribution approximation: under a density-ratio calibration
condition on acceptance probabilities and bounded teacher-replacement error, we bound the approximation error between the resulting mixed
trajectory distribution and a teacher-preferred target. On Web Q&A, OVD achieves 41.09% average EM with teacher feedback at inference,
exceeding the strongest evaluated baseline by 5.89 percentage points. On AMC23, OVD-FR improves accuracy over RLVR by 10.0 percentage points
(52.5% to 62.5%) after 600 training steps on 128 problems. Further experiments suggest that retaining student-generated prefixes helps
preserve exploration and mitigate trajectory-level entropy collapse. OVD also improves training efficiency: resampling selected suffixes
rather than entire responses reduces mean per-step training time by 10.2% in the 128-problem setting. Project page: https://menik1126.github.io/ovd-project-page/.
Original source
This story was published by arXiv cs.CL and written by Jing Xiong, Hui Shen, Shansan Gong, Yuxin Cheng, Jianghan Shen, Chaofan Tao, Haochen Tan, Haoli Bai, Lifeng Shang, Ngai Wong. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on arxiv.org


