From Evolutionary Operation to Parallel Direct Search: Pattern Search Algorithms for Numerical Optimization

G.E.P. Box’s seminal suggestions for Evolutionary Operation led other statisticians to propose algorithms for numerical optimization that rely exclusively on the direct comparison of function values. These contributions culminated in the development of the widely used simplex algorithm of Nelder and Mead. Recent examination of these popular methods by the numerical optimization community has produced new insights. Numerical experiments and carefully constructed examples have revealed that the Nelder-Mead algorithm may be unreliable even in fairly simple situations. In contrast, many of the original methods, which we collectively describe as pattern searches, are guaranteed to converge to a stationary point of the objective function under conventional nonlinear programming assumptions. In addition, the structure of these algorithms is such that they are easily parallelized. We will briefly survey the history of pattern search methods and explicate their common structure, pointing out the key features that the Nelder-Mead simplex algorithm lacks. We will close with some practical suggestions for using pattern searches in serial and in distributed computing environments.