Towards optimal stochastic alternating direction method of multipliers: Supplementary material

1. The strongly convex case 1.1. Proof of Lemma 1 Lemma 1. Let f be µ-strongly convex, and let x k+1 , y k+1 and λ k+1 be computed as per Alg. 2. For all x ∈ X and y ∈ Y, and w ∈ Ω, it holds for k ≥ 0 that f (x k) − f (x) + h(y k+1) − h(y) + ⟨w k+1 − w, F (w k+1)⟩ ≤ η k 2 ∥g k ∥ 2 2 − µ 2 ∆ k + 1 2η k [∆ k − ∆ k+1 ] + β 2 [A k − A k+1 ] + 1 2β [L k − L k+1 ] + ⟨δ k , x k − x⟩. By the strong convexity of f , we have f (x k) − f (x) ≤ ⟨f ′ (x k), x k − x⟩ − µ 2 ∥x k − x∥ 2 2. (2) As before, using δ k = f ′ (x k) − g k ; but this time we split the f ′ (x k) term differently: ⟨f ′ Now for the first part, we just follow the derivation of (Ouyang et al., 2013), before it comes to the critical difference, namely inequality (9). However, for the reader's convenience we include all the details below. From the optimality condition of Line 2, it follows that ⟨g k + βA T (Ax k+1 + By k − b) − A T λ k + η −1 k (x k+1 − x k), x − x k+1 ⟩ ≥ 0, ∀x ∈ X. Rearranging this inequality, we obtain ⟨g k , x k+1 − x⟩ ≤ ⟨βA T (Ax k+1 + By k − b) − A T λ k , x − x k+1 ⟩ + 1 η k ⟨x k+1 − x k , x − x k+1 ⟩, so that a rearrangement similar to (20) yields ⟨g k , x k+1 − x⟩ ≤ ⟨βA T (Ax k+1 + By k − b) − A T λ k , x − x k+1 ⟩ + 1 2η k [ ∥x − x k ∥ 2 2 − ∥x − x k+1 ∥ 2 2 − ∥x k+1 − x k ∥ 2 2 ] .