Faster Adaptive Federated Learning

Federated learning has attracted increasing attention with the emergence of distributed data. While extensive federated algorithms have been proposed for non-convex problem, in practice still faces numerous challenges, such as large training iterations to converge since sizes models and datasets keep increasing, lack adaptivity by SGD-based model updates. Meanwhile, study adaptive methods is scarce existing works either a complete theoretical convergence guarantee or slow sample complexity. In this paper, we propose an efficient algorithm (i.e., FAFED) based on momentum-based variance reduced technique cross-silo FL. We first explore how design FL setting. By providing counter-example, prove that simple combination could lead divergence. More importantly, provide analysis our method reach best-known samples O(epsilon(-3)) O(epsilon(-2)) communication rounds find epsilon-stationary point without batches. The experimental results language modeling task image classification heterogeneous data demonstrate efficiency algorithms.