Intelligents et de Robotique Hierarchical & Factored Reinforcement Learning

This thesis is accomplished in the context of the industrial simulation domain that addresses the problems of modelling of human behavior in military training and civil security simulations. The aim of this work is to solve large stochastic and sequential decision making problems in the Markov Decision Process (MDP) framework using Reinforcement Learning methods for learning and planning under uncertainty. The Factored Markov Decision Process (FMDP) framework is a standard representation for sequential decision problems under uncertainty where the state is represented as a collection of random variables. Factored Reinforcement Learning (FRL) is an Modelbased Reinforcement Learning approach to FMDPs where the transition and reward functions of the problem are learned under a factored form. As a first contribution of this thesis, we show how to model in a theoretically well-founded way the problems where some combinations of state variable values may not occur, giving rise to what we call impossible states. Furthermore, we propose a new heuristics that considers as impossible the states that have not been seen so far. We derive an algorithm whose improvement in performance with respect to the standard approach is illustrated through benchmark experiments on MAZE6 and BLOCKS WORLD problems. Besides, following the example of FMDPs, a Hierarchical MDP (HMDP) is based on the idea of factorization, but brings that idea on a new level. From state factorization of FMDPs, HMDP can make profit of task factorization, where a set of similar situations (defined by their goals) are represented by a partially defined set of independent subtasks. In other words, it is possible to simplify a problem by splitting it into smaller problems that are easier to solve individually, but also reuse the subtasks in order to speed up the global search of a solution. This kind of architecture can be eficiently represented using the options framework by including temporally extended courses of actions. The second contribution of this thesis introduces TeXDYNA, an algorithm designed to solve large MDPs with unknown structure by integrating hierarchical abstraction techniques of Hierarchical Reinforcement Learning (HRL) and factorization techniques