AdaDelay: Delay Adaptive Distributed Stochastic Convex Optimization

We study distributed stochastic convex optimization under the delayed gradient model where theserver nodes perform parameter updates, while the worker nodes compute stochastic gradients. Wediscuss, analyze, and experiment with a setup motivated by the behavior of real-world distributedcomputation networks, where the machines are differently slow at different time. Therefore, we allowthe parameter updates to be sensitive to the actual delays experienced, rather than to worst-casebounds on the maximum delay. This sensitivity leads to larger stepsizes, that can help gain rapidinitial convergence without having to wait too long for slower machines, while maintaining the sameasymptotic complexity. We obtain encouraging improvements to overall convergence for distributedexperiments on real datasets with up to billions of examples and features.