Unconstrained Derivative-Free Optimization by Successive Approximation

We present an algorithmic framework for unconstrained derivative-free optimization based on dividing the search space in regions (partitions). Every partition is assigned a representative point. The representative points form a grid. A piece-wise constant approximation to the function subject to optimization is constructed using a partitioning and its corresponding grid. The convergence of the framework to a stationary point of a continuously differentiable function is guaranteed under mild assumptions. The proposed framework is appropriate for upgrading heuristics that lack mathematical analysis into algorithms that guarantee convergence to a local minimizer. A convergent variant of the Nelder-Mead algorithm that conforms to the given framework is constructed. The algorithm is compared to two previously published convergent variants of the NM algorithm. The comparison is conducted on the Moré-Garbow-Hillstrom set of test problems and on four variably dimensional functions with dimension up to 100. The results of the comparison show that the proposed algorithm outperforms both previously published algorithms.