Predictive Property of Hidden Representations in Recurrent Neural Network Language Models

The hidden representation of a recurrent neural network language model (RNNLM) is regarded as a summary of the past input sequence. In this study, we propose that the hidden representation also consists of the expectation about upcoming inputs. A RNNLM is originally trained to predict the next word or character, but we experimentally discover, even for an unmodified RNNLM, the farther sequences can also be predicted given the activation of the hidden neurons. This property makes the hidden activation a summary of the local context covering both the past and the near future, which may benefit some language processing tasks which did not previously take advantage of language models. Dimensionality reduction approach is also briefly considered to facilitate the practical application of the predictive property.