Improving on the reinforcement learning of coordination in cooperative multi-agent systems

We report on an investigation of reinforcement learning techniques for the learning of coordination in cooperative multiagent systems. These techniques are variants of Q-learning (Watkins, 1989) that are applicable to scenarios where mutual observation of actions is not possible. To date, reinforcement learning approaches for such independent agents did not guarantee convergence to the optimal joint action in scenarios with high miscoordination costs. We improve on previous results (Claus and Boutilier, 1998) by demonstrating that our extension causes the agents to converge almost always to the optimal joint action even in these difficult cases.