Predictive Q-routing: a Memory-based Reinforcement Learning Approach to Adaptive Traac Control

In this paper, we propose a memory-based Q-learning algorithm called predictive Q-routing (PQ-routing) for adaptive traac control. We attempt to address two problems encountered in Q-routing (Boyan & Littman, 1994), namely, the inability to ne-tune routing policies under low network load and the inability to learn new optimal policies under decreasing load conditions. Unlike other memory-based reinforcement learning algorithms in which memory is used to keep past experiences to increase learning speed, PQ-routing keeps the best experiences learned and reuses them by predicting the traac trend. The eeectiveness of PQ-routing has been veriied under various network topologies and traac conditions. Simulation results show that PQ-routing is superior to Q-routing in terms of both learning speed and adaptability.