Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reduction

Asynchronous Q-learning aims to learn the optimal action-value function (or Q-function) of a Markov decision process (MDP), based on single trajectory Markovian samples induced by behavior policy. Focusing $\gamma $ -discounted MDP with state space notation="LaTeX">$\mathcal {S}$ and action {A}$ , we demonstrate that notation="LaTeX">$\ell _{\infty }$ -based sample complexity classical asynchronous — namely, number needed yield an entrywise notation="LaTeX">$\varepsilon -accurate estimate Q-function is at most order notation="LaTeX">$\frac {1}{ \mu _{\mathsf {min}}(1-\gamma)^{5}\varepsilon ^{2}}+ \frac { t_{\mathsf {mix}}}{ {min}}(1-\gamma)}$ up some logarithmic factor, provided proper constant learning rate adopted. Here, notation="LaTeX">$t_{\mathsf {mix}}$ notation="LaTeX">$\mu {min}}$ denote respectively mixing time minimum state-action occupancy probability trajectory. The first term this bound matches in synchronous case independent drawn from stationary distribution second reflects cost taken for empirical reach steady state, which incurred very beginning becomes amortized as algorithm runs. Encouragingly, above improves upon state-of-the-art result factor least notation="LaTeX">$|\mathcal {S}||\mathcal {A}|$ all scenarios, {mix}}|\mathcal any sufficiently small accuracy level . Further, scaling effective horizon {1}{1-\gamma can be improved means variance reduction.