Fast-and-Light Stochastic ADMM Appendix

In the second equality, we use (1). In the third equality, we use E‖xi−Exi‖ = E‖xi‖−‖Exi‖. In the second inequality, we use ‖a+b‖ ≤ 2‖a‖+2‖b‖. In the last inequality, we employ the following fact [Xiao and Zhang, 2014]: 1 n ∑n i=1 ‖∇fi(x)− ∇fi(x∗)‖2 ≤ 2Lmax ( f(x)− f(x∗)−∇f(x∗) (x− x∗) ) . .2 Proof of Theorem 1 First, we introduce the following Lemma. Lemma 1. u∗ = − 1 ρ (A T )∇f(x∗). Proof. Consider (4) as a linear system Au = − 1 ρ∇f(x∗) for a random variable u. By [James, 1978], the solutions are given by