CS229 Final Project: Language Grounding in Minecraft with Gated-Attention Networks

A key question in language understanding is the problem of language grounding – how do symbols such as words get their meaning? We examine this question in the context of task oriented language grounding in gameplay. In order to perform tasks and challenges specified by natural language instructions, agents need to extract semantically meaningful representations of language and map it to the visual elements of their scene and into actions in the environment. This is often referred to as task-oriented language grounding. In this project, we propose to directly map raw visual observations and text input into actions for instruction execution, using an end-to-end trainable neural architecture. The model synthesizes image and text representations using Gated-Attention mechanisms and learns a policy using Stein Variational policy gradients to execute the natural language instruction. We evaluate our method in the Minecraft environment to the problem of retrieving items in rooms and mazes and show improvements over supervised and common reinforcement learning algorithms.