Exploring Multiple Solutions in Graphical Models by Cluster Sampling

This paper presents a novel Markov Chain Monte Carlo (MCMC) inference algorithm called C4 – Clustering with Cooperative and Competitive Constraints for computing multiple solutions from posterior probabilities defined on graphical models, including Markov random fields (MRF), conditional random fields (CRF) and hierarchical models. The graphs may have both positive and negative edges for cooperative and competitive constraints. C4 is a probabilistic clustering algorithm in the spirit of Swendsen-Wang [34]. By turning the positive edges on/off probabilistically, C4 partitions the graph into a number of connected components (ccp’s) and each ccp is a coupled sub-solution with nodes connected by positive edges. Then by turning the negative edges on/off probabilistically, C4 obtains composite ccp’s (called cccp’s) with competing ccp’s connected by negative edges. At each step C4 flips the labels of all nodes in a cccp so that nodes in each ccp keep the same label while different ccp’s are assigned different labels to observe both positive and negative constraints. Thus the algorithm can jump between multiple competing solutions (or modes of the posterior probability) in a single or a few steps. It computes multiple distinct solutions to preserve the intrinsic ambiguities and avoids premature commitments to a single solution that may not be valid given later context. C4 achieves a mixing rate faster than existing MCMC methods, such as various Gibbs samplers [15], [26] and Swendsen-Wang cuts [2], [34]. It is also more “dynamic” than common optimization method such as ICM [3], LBP [21], [37], and graph cuts [4], [20]. We demonstrate the C4 algorithm in line drawing interpretation, scene labeling, and object recognition.