The Thing that we Tried Didn't Work very Well: Deictic Representation in Reinforcement Learning

Most reinforcement learning methods oper­ ate on propositional representations of the world state. Such representations are of­ ten intractably large and generalize poorly. Using a deictic representation is believed to be a viable alternative: they promise gener­ alization while allowing the use of existing reinforcement-learning methods. Yet, there are few experiments on learning with deic­ tic representations reported in the literature. In this paper we explore the effectiveness of two forms of deictic representation and a naive propositional representation in a sim­ ple blocks-world domain. We find, empiri­ cally, that the deictic representations actu­ ally worsen learning performance. We con­ clude with a discussion of possible causes of these results and strategies for more effective learning in domains with objects.