
YG
Yaozhong Gan, Renye Yan, Xiaoyang Tan, Zhe Wu, Junliang Xing
· 1 min read
ResearcharXiv cs.LG
Transductive Off-policy Proximal Policy Optimization
arXiv:2406.03894v3 Announce Type: replace
Abstract: Proximal Policy Optimization (PPO) is a popular model-free reinforcement learning algorithm, esteemed for its simplicity and efficacy. However, due to its inherent on-policy nature, its proficiency in harnessing data from disparate policies is constrained. This paper introduces a novel off-policy extension to the original PPO method, christened Transductive Off-policy PPO (ToPPO). Herein, we provide theoretical justification for incorporating off-policy data in PPO training and prudent guidelines for its safe application. Our contribution includes a novel formulation of the policy improvement lower bound for prospective policies derived from off-policy data, accompanied by a computationally efficient mechanism to optimize this bound, underpinned by assurances of monotonic improvement. Comprehensive experimental results across six representative tasks underscore ToPPO's promising performance.
Original source
This story was published by arXiv cs.LG and written by Yaozhong Gan, Renye Yan, Xiaoyang Tan, Zhe Wu, Junliang Xing. SyncAI.news shows a preview; the complete article is on the publisher's site.
Read the full story on arxiv.org


