A New Learning Algorithm for Mean Field Boltzmann Machines

We present a new learning algorithm for Mean Field Boltzmann Machines based on the contrastive divergence optimization criterion. In addition to minimizing the divergence between the data distribution and the equilibrium distribution that the network believes in, we maximize the divergence between one-step reconstructions of the data and the equilibrium distribution. This eliminates the need to estimate equilibrium statistics, so we do not need to approximate the multimodal probablility distribution of the free network with the unimodal mean field distribution. We test the learning algorithm on the classification of digits. A New Learning Algorithm for Mean Field Boltzmann Machines Max Welling G.E. Hinton Gatsby Unit 1 Boltzmann Machines The stochastic Boltzmann machine (BM) is a probabilistic neural network of symmetrically connected binary units taking values f0; 1g (Ackley, Hinton & Sejnowski, 1985). The variant used for unsupervised learning consists of a set of visible units v, which are clamped to the data v1:N , and a set of hidden units h, which allow the modelling of higher order statistics of the data. We may define the energy E of the system at a particular state fv;hg to be, E(v;h) = ( 1 2 v T Vv + 1 2 h T Wh+ vJh) (1) where we have added one unit with value always 1, whose weights to all other units represent the biases. In terms of the energy, the probability distribution of the system can be written as,