Maximum Reconstruction Estimation for Generative Latent-Variable Models

Generative latent-variable models are important for natural language processing due to their capability of providing compact representations of data. As conventional maximum likelihood estimation (MLE) is prone to focus on explaining irrelevant but common correlations in data, we apply maximum reconstruction estimation (MRE) to learning generative latent-variable models alternatively, which aims to find model parameters that maximize the probability of reconstructing the observed data. We develop tractable algorithms to directly learn hidden Markov models and IBM translation models using the MRE criterion, without the need to introduce a separate reconstruction model to facilitate efficient inference. Experiments on unsupervised part-of-speech induction and unsupervised word alignment show that our approach enables generative latent-variable models to better discover intended correlations in data and outperforms maximum likelihood estimators significantly. Introduction The need to learn latent structures from unlabeled data arises in many different problems in natural language processing (NLP), including part-of-speech (POS) induction (Merialdo 1994; Johnson 2007), word alignment (Brown et al. 1993; Vogel, Ney, and Tillmann 1996), syntactic parsing (Klein and Manning 2004; Smith and Eisner 2005), and semantic parsing (Poon and Domingos 2009). Generative latentvariable models such as hidden Markov models (HMMs) and latent-variable probabilistic context-free grammars (LPCFGs) have been widely used for unsupervised structured prediction due to their capability of providing compact representations of data. Maximum likelihood estimation (MLE) is the standard criterion for training generative latent-variable models: maximizing the likelihood of observed data by marginalizing over latent variables, typically via the Expectation Maximization (EM) algorithm. However, recent studies have revealed that MLE suffers from a significant problem: it may guide the model to focus on explaining irrelevant but common correlations in the data (Ganchev et al. 2010). For ex∗ Yang Liu is the corresponding author: liuyang2011@ tsinghua. edu.cn. Copyright c © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. ample, in unsupervised POS induction, estimating tag probabilities are often prone to be disrupted by high-frequency words such as “,” and “is” (see Table 3). In this work, we introduce maximum reconstruction estimation (MRE) (Hinton, Osindero, and Teh 2006; Vincent et al. 2008; Bengio 2009; Vincent et al. 2010; Socher et al. 2011; Ammar, Dyer, and Smith 2014; Alain et al. 2015) for learning generative latent-variable models. The basic idea is to circumvent irrelevant but common correlations by maximizing the probability of reconstructing observed data. The new objective is based on an encoding-reconstruction framework: first generating a latent structure conditioned on the observed data (encoding) and then re-generating the observation based on the latent structure (reconstruction). Our approach has the following advantages: 1. Direct learning of model parameters. As we are only interested in learning the parameters of data distribution (i.e., P (x;θ)), it is challenging to define the reconstruction distribution (i.e., P (x|x;θ)) using data distribution parameters (i.e., θ) due to the intricate dependencies between sub-structures in generative latent-variable models. While previous work has to maintain separate sets of model parameters for encoding and reconstruction to facilitate efficient inference (Ammar, Dyer, and Smith 2014), our approach is capable of directly learning intended model parameters. 2. Tractable inference. Our approach keeps the same tractability of learning generative latent-variable models as its MLE counterpart. In this work, we apply MRE to learning two classic generative latent-variable models (i.e., HMMs and IBM translation models) and design efficient dynamic programming algorithms to calculate expectations exactly. We evaluate our approach on unsupervised POS induction and unsupervised word alignment. Experiments show that MRE enables generative latent-variable models to better discover intended correlations in data and thus leads to significant improvements over MLE. Maximum Reconstruction Estimation Let x be an observation, z be a latent structure, and P (x;θ) be a generative latent-variable model parameterized by θ: P (x;θ) = ∑