
JX
Jingqiao Xiu, Yicong Li
· 1 min read
ResearcharXiv cs.LG
Reinforcing Few-Step Diffusion Models via Shadow Posterior Matching
arXiv:2611.25998v1 Announce Type: new
Abstract: Few-step diffusion models are widely deployed for their efficiency. However, reinforcing them with black-box rewards, as is routine for multi-step models, faces two structural challenges. First, a few-step generator is a sampler rather than a score model, so its velocity is not the forward velocity of the generated distribution. Second, it must be anchored to the undistilled teacher, which shares no transitions with it. Existing few-step RL methods handle the latter with a marginal KL to the teacher, but each pays for the former in its reward credit. We propose Shadow Posterior Matching (SPM), which renoises generated samples into shadow inputs and obtains the credit from their shadow posterior. From the KL-regularized reward objective, we derive a regression loss that turns reward values into a reward direction while anchoring the generator's outputs to the teacher and aligning the shadow inputs with the deployed inputs. Applied to 4-step generators, SPM achieves higher rewards with less reward hacking than prior few-step RL methods.
Original source
This story was published by arXiv cs.LG and written by Jingqiao Xiu, Yicong Li. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on arxiv.org


