Fully asynchronous policy evaluation in distributed reinforcement learning over networks

This paper proposes a fully asynchronous scheme for the policy evaluation problem of distributed reinforcement learning (DisRL) over directed peer-to-peer networks. Without waiting any other node network, each can locally update its value function at time using (possibly delayed) information from neighbors. is in sharp contrast to gossip-based where pair nodes concurrently update. Even though setting involves difficult multi-timescale decision problem, we design novel incremental aggregated gradient (IAG) based algorithm and develop push–pull augmented graph approach prove exact convergence linear rate O(ck) c∈(0,1) k total number updates within entire network. Finally, numerical experiments validate that our method speeds up linearly with respect nodes, robust straggler nodes.