Hugging Face Journal Club: Direct On-Policy Distillation

The Hugging Face research team discusses the paper "Weak-to-Strong Generalization via Direct On-Policy Distillation" which proposes a cheap way to transfer the benefits of reinforcement learning from small models to much larger ones. Instead of imitating the smaller model directly, it measures how RL changed the small model’s policy and uses that change as a dense reward signal to train the larger model. The result is a form of weak-to-strong generalization that can match or outperform direct RL on the larger model at a fraction of the compute cost. Paper link: