We present a method for distributing collapsed Gibbs sampling over multiple processors that is simple, statistically correct, and memory efficient. The method uses blocked sampling, dividing the training data into relatively large sized blocks, and distributing the sampling of each block over multiple processors. At the end of each parallel run, MetropolisHastings rejection sampling is performe...