Two-Timescale Q-Learning with an Application to Routing in Communication Networks

We propose two variants of the Q-learning algorithm that (both) use two timescales. One of these updates Q-values of all feasible state-action pairs at each instant while the other updates Q-values of states with actions chosen according to the ‘current’ randomized policy updates. A sketch of convergence of the algorithms is shown. Finally, numerical experiments using the proposed algorithms for routing on different network topologies are presented and performance comparisons with the regular Q-learning algorithm are shown. Index Terms Q-learning based algorithms, Markov decision processes, two-timescale stochastic approximation, simultaneous perturbation stochastic approximation (SPSA), normalized Hadamard matrices.