SyncAI.news, a Varaisys broadcasting
Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control
CM

Christian Moya, Elliott Thornley, Guang Lin

· 1 min read

ResearcharXiv cs.AI

Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control

arXiv:2609.35677v1 Announce Type: new Abstract: In reinforcement learning with verifiable rewards (RLVR), imperfect verifiers can reward incorrect responses, creating opportunities for reward hacking. Using gradient flow with a fixed verifier, we characterize the conditions under which reward rises while correctness falls. We then show that the observations available during RLVR are, in general, insufficient to detect or identify accepted errors, or to guarantee their reduction without sacrificing correct responses. To address this limit, we construct a correction using additional feedback about correctness from audits. This correction achieves \emph{selective control}: at the current policy, it lowers the probability of accepted errors and raises that of correct responses, provided it outweighs the pressure toward errors from verifier reward. Experiments with log linear and neural contextual bandits and with a language model support the analysis and show that selective control under partial auditing reduces accepted errors while increasing correctness.

Original source

This story was published by arXiv cs.AI and written by Christian Moya, Elliott Thornley, Guang Lin. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News