Optimal rates for first-order stochastic convex optimization under Tsybakov noise condition

We focus on the problem of minimizing a convex function f over a convex set S given T queries to a stochastic first order oracle. We argue that the complexity of convex minimization is only determined by the rate of growth of the function around its minimizer xf,S , as quantified by a Tsybakov-like noise condition. Specifically, we prove that if f grows at least as fast as ‖x − xf,S‖ around its minimum, for some κ > 1, then the optimal rate of learning f(xf,S) is Θ(T − κ 2κ−2 ). The classic rate Θ(1/ √ T ) for convex functions and Θ(1/T ) for strongly convex functions are special cases of our result for κ→∞ and κ = 2, and even faster rates are attained for κ < 2. We also derive tight bounds for the complexity of learning xf,S , where the optimal rate is Θ(T− 1 2κ−2 ). Interestingly, these precise rates for convex optimization also characterize the complexity of active learning and our results further strengthen the connections between the two fields, both of which rely on feedback-driven queries.