Graphical Newton for Huge-Block Coordinate Descent on Sparse Graphs

Block coordinate descent (BCD) methods are often very effective for optimization problems where dependencies between variables are sparse. These methods can make a substantial amount of progress by applying Newton’s method to update a block of variables, but this leads to an iteration cost of O(|b|) in terms of the block size |b|. In this paper, we show how to use message-passing to compute the Newton step in O(|b|) when the block has a forest-structured dependency. Consequently, this allows us to update huge blocks for sparse problems, resulting in significant numerical improvements over existing approaches. We also present a greedy approach for selecting forest-structured blocks.