Hierarchical Imitation and Reinforcement Learning
نویسندگان
چکیده
We study the problem of learning policies over long time horizons. We present a framework that leverages and integrates two key concepts. First, we utilize hierarchical policy classes that enable planning over different time scales, i.e., the high level planner proposes a sequence of subgoals for the low level planner to achieve. Second, we utilize expert demonstrations within the hierarchical action space to dramatically reduce cost of exploration. Our framework is flexible and can incorporate different combinations of imitation learning (IL) and reinforcement learning (RL) at different levels of the hierarchy. Using long-horizon benchmarks, including Montezuma’s Revenge, we empirically demonstrate that our approach can learn significantly faster compared to hierarchical RL, and can be significantly more labeland sample-efficient compared to flat IL. We also provide theoretical analysis of the labeling cost for certain instantiations of our framework.
منابع مشابه
Learning by Imitation, Reinforcement and Verbal Rules in Problem Solving Tasks
Learning by imitation is a powerful process for acquiring new knowledge, but there has been little research exploring imitation’s potential in the problem solving domain. Classical problem solving techniques tend to center around reinforcement learning, which requires significant trial-and-error learning to reach successful goals and problem solutions. Heuristics, hints, and reasoning by analog...
متن کاملLearning by imitation, by reinforcement and by verbal rules in problem solving
Learning by imitation is a powerful process for acquiring new knowledge, but there has been little research exploring imitation’s potential service to the problem-solving domain. Classical problem-solving techniques tend to center around reinforcement learning, which requires significant trial-and-error learning to reach successful goals and problem solutions. Heuristics, hints, and reasoning b...
متن کاملA Constructive Connectionist Approach Towards Continual Robot Learning
This work presents an approach for combining reinforcement learning, learning by imitation, and incremental hierarchical development. The approach is used in a realistic simulated mobile robot that learns to perform a navigation task by imitating the movements of a teacher and then continues to learn by receiving reinforcement. The behaviours of the robot are represented as sensation-action rul...
متن کاملContinual Robot Learning withConstructive Neural
In this paper, we present an approach for combining reinforcement learning, learning by imitation, and incremental hierarchical development. We apply this approach to a realistic simulated mobile robot that learns to perform a navigation task by imitating the movements of a teacher and then continues to learn by receiving reinforcement. The behaviours of the robot are represented as sensation-a...
متن کاملHierarchical Functional Concepts for Knowledge Transfer among Reinforcement Learning Agents
This article introduces the notions of functional space and concept as a way of knowledge representation and abstraction for Reinforcement Learning agents. These definitions are used as a tool of knowledge transfer among agents. The agents are assumed to be heterogeneous; they have different state spaces but share a same dynamic, reward and action space. In other words, the agents are assumed t...
متن کامل