This paper argues that many visual scenes are based on a ”Manhattan” three-dimensional grid which imposes regularities on the image statistics. We construct a Bayesian model which implements this assumption and estimates the viewer orientation relative to the Manhattan grid. For many images, these estimates are good approximations to the viewer orientation (as estimated manually by the authors)...