
SY
Simon Yu, Nicholas Tomlin, Marwa Abdulhai, Ximing Lu, Derek Chong, Abe Hou, Dilara Soylu, Sergey Levine, Christopher D. Manning, Weiyan Shi
· 1 min read
ResearcharXiv cs.CL
One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL
arXiv:2608.12253v3 Announce Type: replace
Abstract: Multi-agent reinforcement learning (MARL) for human-AI interaction usually trains the agent against a single large language model (LLM) that simulates the user. We show that agents trained this way can systematically fail to generalize, and we trace the failure to simulator collapse: the simulator LLM is mode-collapsed, and a policy trained against it learns narrow strategies that exploit the simulator's mode behavior; these strategies transfer poorly to unseen simulators and to real users. We formalize the collapse and propose two solutions. Verbalized Sampling, applied at inference time, samples each simulator reply from a verbalized response distribution. Co-Training, applied during training similar to self-play, co-evolves the simulator as the policy, so the policy cannot exploit the mode behavior. A population variant samples the simulator for each rollout from a pool of recent checkpoints. We validate both solutions on three benchmarks: Persuasion for Good, tau^2-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions also preserve the policy diversity that collapses under single-simulator RL. Overall, our findings on simulator collapse point to diverse simulators as a requirement for RL agents that serve real users. We release SCOPE, an open-source framework for Population Co-Training in multi-agent RL.
Original source
This story was published by arXiv cs.CL and written by Simon Yu, Nicholas Tomlin, Marwa Abdulhai, Ximing Lu, Derek Chong, Abe Hou, Dilara Soylu, Sergey Levine, Christopher D. Manning, Weiyan Shi. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on arxiv.org


