Distributed statistical learning problems arise commonly when dealing with large datasets. In this setup, datasets are partitioned over machines, which compute locally, and communicate short messages. Communication is often the bottleneck. paper, we study one-step iterative weighted parameter averaging in linear models under data parallelism. We do regression on each machine, send results to a ...

We study random design linear regression with no assumptions on the distribution of covariates and a heavy-tailed response variable. In this distribution-free setting, we show that boundedness conditional second moment given is necessary sufficient condition for achieving nontrivial guarantees. As starting point, prove an optimal version classical in-expectation bound truncated least squares es...