Behaviour-Based Reinforcement Learning

Although behaviour-based robotics has been successfully used to develop autonomous mobile robots up to a certain point, further progress may require the integration of a learning model into the behaviour-based framework. Reinforcement learning is a natural candidate for this because it seems well suited to the problems faced by autonomous agents. However, previous attempts to use reinforcement learning in behaviour-based mobile robots have been simple combinations of these two methodologies rather than full integrations, and have suffered from severe scaling problems that appear to make them infeasible. Furthermore, the implicit assumptions that form the basis of reinforcement learning theory were not developed with the problems faced by autonomous agents in complex environments in mind. This dissertation introduces a model of reinforcement learning that is designed specifically for use in behaviour-based robots, taking the conditions faced by situated agents into account. The model layers a distributed and asynchronous reinforcement learning algorithm over a learned topological map and standard behavioural substrate to create a reinforcement learning complex. The topological map creates a small and task-relevant state space that aims to make reinforcement learning feasible, while the distributed and asynchronous nature of the model makes it compatible with behaviour-based design principles. The model is then validated through an experiment that requires a mobile robot to perform puck foraging in three separate artificial arenas. The development of Dangerous Beans, a mobile robot that is capable of building a distributed topological map of its environment and performing reinforcement learning over it is described, along with the results of its use to test three control strategies (random decision making, a standard reinforcement learning algorithm layered on top of a topological map, and the full model developed in this dissertation) in the arenas. The results show that the model developed in this dissertation is able to learn rapidly in a real environment, and outperforms both the random strategy and the layered standard reinforcement learning algorithm. Following this, a discussion of the implications of these results is given, which suggests that situated learning and the integration of behaviour-based methods and layered learning models merit further study.