Simple and cumulative regret for continuous noisy optimization

Various papers have analyzed the noisy optimization of convex functions. This analysis has been made according to several criteria used to evaluate the performance of algorithms: uniform rate, simple regret and cumulative regret. We propose an iterative optimization framework, a particular instance of which, using Hessian approximations, provably (i) reaches the same rate as Kiefer-Wolfowitz algorithm when the noise has constant variance (ii) reaches the same rate as Evolution Strategies when the noise variance decreases quadratically as a function of the simple regret (iii) reaches the same rate as Bernstein-races optimization algorithms when the noise variance decreases linearly as a function of the simple regret.