SyncAI.news, a Varaisys broadcasting
Reinforcing Few-Step Diffusion Models via Shadow Posterior Matching
JX

Jingqiao Xiu, Yicong Li

· 1 min read

ResearcharXiv cs.LG

Reinforcing Few-Step Diffusion Models via Shadow Posterior Matching

arXiv:2611.25998v1 Announce Type: new Abstract: Few-step diffusion models are widely deployed for their efficiency. However, reinforcing them with black-box rewards, as is routine for multi-step models, faces two structural challenges. First, a few-step generator is a sampler rather than a score model, so its velocity is not the forward velocity of the generated distribution. Second, it must be anchored to the undistilled teacher, which shares no transitions with it. Existing few-step RL methods handle the latter with a marginal KL to the teacher, but each pays for the former in its reward credit. We propose Shadow Posterior Matching (SPM), which renoises generated samples into shadow inputs and obtains the credit from their shadow posterior. From the KL-regularized reward objective, we derive a regression loss that turns reward values into a reward direction while anchoring the generator's outputs to the teacher and aligning the shadow inputs with the deployed inputs. Applied to 4-step generators, SPM achieves higher rewards with less reward hacking than prior few-step RL methods.

Original source

This story was published by arXiv cs.LG and written by Jingqiao Xiu, Yicong Li. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News