Predictive Random Fields: Latent Variable Models Fit by Multiway Conditional Probability with Applications to Document Analysis

We introduce predictive random fields, a framework for learning undirected graphical models based not on joint, generative likelihood, or on conditional likelihood, but based on a product of several conditional likelihoods each relying on common sets of parameters and predicting different subsets of variables conditioned on other subsets. When applied to models with latent variables, such as the Harmonium, this approach results in powerful clustering models that combine the advantages of conditional random fields with the unsupervised clustering ability of popular topic models, such as latent Dirichlet allocation and its successors. We present new algorithms for parameter estimation based on contrastive divergence. Experimental results show significant improvement in inferring hidden document categories, and learning models of authors, words, topics and time.