Zeroth-order optimization with orthogonal random directions

We propose and analyze a randomized zeroth-order optimization method based on approximating the exact gradient by finite differences computed in set of orthogonal random directions that changes with each iteration. A number previously proposed methods are recovered as special cases including spherical smoothing, coordinate descent, well discretized descent. Our main contribution is proving convergence guarantees rates under different parameter choices assumptions. In particular, we consider convex objectives, but also possibly non-convex objectives satisfying Polyak-Łojasiewicz (PL) condition. Theoretical results complemented illustrated numerical experiments.