SyncAI.news, a Varaisys broadcasting
Gradient-Based Co-Design of Systems and Reinforcement Learning Policies with Differentiable Simulators
AS

Akhil S. Anand, Xun Shen, Marc Peter Deisenroth, Sebastien Gros

· 1 min read

ResearcharXiv cs.LG

Gradient-Based Co-Design of Systems and Reinforcement Learning Policies with Differentiable Simulators

arXiv:2611.19329v1 Announce Type: new Abstract: Co-design is the problem of jointly optimizing the design of a system and the policy that operates it. Neither can be chosen independently: a system design is only as good as the policy trained for it, and the optimal policy changes with the design. When that policy is learned by reinforcement learning (RL), evaluating a candidate design requires a full policy training run, and the design is usually optimized by black-box search such as Bayesian optimization (BO) or evolution strategies (ES). These methods scale poorly with the design dimension and ignore the differentiable structure of modern simulators and, in particular, the sensitivity of the optimal policy with respect to the design. We instead derive a gradient-based co-design method, where the design gradient decomposes into a direct effect of changing the design at a fixed policy and an indirect effect through the design-dependent optimal policy. The indirect effect requires the sensitivity of the converged policy to the design, which we obtain by applying the implicit function theorem to the stationarity condition of the policy gradient. Based on this sensitivity, we develop a proof-of-concept co-design algorithm with stochastic value gradient (SVG)-style policy optimization, prove that it converges to an approximately stationary design, and demonstrate on a differentiable networked-system benchmark with high-dimensional design spaces that it consistently outperforms BO and ES.

Original source

This story was published by arXiv cs.LG and written by Akhil S. Anand, Xun Shen, Marc Peter Deisenroth, Sebastien Gros. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News