Gradient Temporal Difference with Momentum: Stability and Convergence

Gradient temporal difference (Gradient TD) algorithms are a popular class of stochastic approximation (SA) used for policy evaluation in reinforcement learning. Here, we consider TD with an additional heavy ball momentum term and provide choice step size parameter that ensures almost sure convergence these asymptotically. In doing so, decompose the iterates into three separate different sizes. We first analyze under one-timescale SA setting using results from current literature. However, case is restrictive more general analysis can be provided by looking at three-timescale decomposition iterates. process conditions stability SA. then prove algorithm convergent our analysis. Finally, evaluate on standard RL problems report improvement performance over vanilla algorithms.