Plans as a Means for Guiding a Reinforcement Learner

The complexity of reinforcement learning problems grows exponentially with the size of the state space, which renders realistic cases unsolvable and underlines the need for guidance. This thesis studies a hybrid agent architecture, in which the toplevel module reuses temporal knowledge in the form of plans that it extracts from a concurrently executing low-level reinforcement learner. The first contribution of this work are significant improvements of the original model and implementation of the agent architecture, resulting in a more effective knowledge extraction and reuse. The second contribution is an extensive exploration of the synergy effects that take place between both layers of the architecture. It is shown that the combination of state abstraction and the reuse of plans as temporal abstraction can lead to a significantly shorter learning time of a reinforcement learning agent. Likewise, the number of decisions to be made by the agent is reduced because a plan is a definite commitment to a course of actions that does not require intermediary reasoning. In addition, we demonstrate that the architecture enables the integration of plans as prior knowledge through a clear and convenient interface. Thus, partial and approximate solutions to the problem can be easily specified to significantly decrease learning time even further. To my beloved wife Agnieszka Beata — for patience, support and encouragement throughout the course of this thesis.