SyncAI.news, a Varaisys broadcasting
Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noise
XQ

Xiaochi Qian, Zixuan Xie, Xinyu Liu, Shangtong Zhang

· 1 min read

ResearcharXiv cs.LG

Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noise

arXiv:2411.13711v2 Announce Type: replace Abstract: This paper establishes, to our knowledge, the first almost sure convergence rate and the first maximal concentration bound with sub-Weibull tails for general contractive stochastic approximation algorithms with Markovian noise. As a corollary, we also obtain convergence rates in $L^p$. Key to our successes is a novel discretization of the mean ODE of stochastic approximation algorithms using intervals with diminishing (instead of constant) length. Applications yield the first almost sure convergence rate for $Q$-learning without count-based learning rates and the first maximal concentration bound for off-policy temporal difference learning, both with Markovian samples.

Original source

This story was published by arXiv cs.LG and written by Xiaochi Qian, Zixuan Xie, Xinyu Liu, Shangtong Zhang. SyncAI.news shows a preview; the complete article is on the publisher's site.

Read the full story on arxiv.org

Similar News