Zeroth-Order Regularized Optimization (ZORO): Approximately Sparse Gradients and Adaptive Sampling

We consider the problem of minimizing a high-dimensional objective function, which may include regularization term, using only (possibly noisy) evaluations function. Such optimization is also called derivative-free, zeroth-order, or black-box optimization. propose new zeroth-order regularized method, dubbed ZORO. When underlying gradient approximately sparse at an iterate, ZORO needs very few function to obtain iterate that decreases achieve this with adaptive, randomized estimator, followed by inexact proximal-gradient scheme. Under novel assumption and various different convex settings, we show (theoretical empirical) convergence rate logarithmically dependent on dimension. Numerical experiments outperforms existing methods similar assumptions, both synthetic real datasets.