On the use of quadratic models in unconstrained minimization without derivatives

Quadratic approximations to the objective function provide a way of estimating first and second derivatives in iterative algorithms for unconstrained minimization. Therefore we address the construction of suitable quadratic models Q by interpolating values of the objective function F . On a typical iteration, the objective function is calculated at the point that minimizes the current quadratic model subject to a trust region bound, and we find that these values of F provide good information for the updating of Q, except that a few extra values are needed occasionally to avoid degeneracy. The number of interpolation points and their positions can be controlled adequately by deleting one of the current points to make room for each new one. An algorithm is described that works in this way. It is applied to an optimization calculation that has between 10 and 160 variables. The numerical results suggest that, if m=2n+1, then the number of evaluations of F is only of magnitude n, where m and n are the number of interpolation conditions of each model and the number of variables, respectively. This success is due to the technique that updates Q. It minimizes the Frobenius norm of the change to ∇Q, subject to the interpolation conditions that have been mentioned. Department of Applied Mathematics and Theoretical Physics, Centre for Mathematical Sciences, Wilberforce Road, Cambridge CB3 0WA, England. March, 2003 (Revised September, 2003). Presented at The First International Conference on Optimization Methods and Software (December, 2002), Hangzhou, China. 1. Least Frobenius norm updating of quadratic models We consider the efficiency and use of quadratic models in iterative algorithms for unconstrained minimization calculations. Let F (x), x∈R, be the objective function whose minimum value is required, let k be the iteration number, and let x∗k be the vector of variables such that F (x ∗ k) is the least calculated value of the objective function so far at the beginning of the k-th iteration. Then the quadratic model of the k-th iteration is a quadratic polynomial Qk(x), x ∈ R , that satisfies Qk(x ∗ k)=F (x ∗ k). Each quadratic model is formed automatically. We will give particular attention to the construction of Qk+1 from Qk, which is done by the k-th iteration. Usually the iteration generates a change to the variables that is based on the approximation Qk(x) ≈ F (x), x∈R . (1.1) Further, in trust region algorithms it is assumed that this approximation is helpful only if x is sufficiently close to x∗k. Specifically, a positive parameter ∆k, called the “trust region radius”, is also available at the beginning of the iteration, and, until termination or some special action is required, the next trial vector of variables is x∗k+dk, where dk is an estimate of the vector d that solves the subproblem Minimize Qk(x ∗ k+d) subject to ‖d‖ ≤ ∆k, (1.2) the vector norm being Euclidean. There are good ways of generating dk that do not require the second derivative matrix ∇Qk to be positive definite (see Moré and Sorensen, 1983, for instance). Then the new function value F (x∗k+dk) may be calculated, and x∗k+1 may be defined by the formula x∗k+1 = { x∗k, F (x ∗ k+dk)≥F (x ∗ k) x∗k+dk, F (x ∗ k+dk)<F (x ∗ k). (1.3) Equation (1.3) provides the best vector of variables so far, but many algorithms set x∗k+1 to x ∗ k+dk only if the reduction F (x ∗ k)−F (x ∗ k+dk) is sufficiently large. Originally this device was introduced to assist proofs of convergence, but now the use of sufficient reductions has become standard practice. In Sections 2 and 3, however, we are going to study a trust region algorithm that retains formula (1.3), because we welcome every decrease that occurs in the value of the objective function. The author does not know of cases where formula (1.3) prevents convergence, provided that the new trust region radius ∆k+1 is chosen carefully. For example, the value ∆k+1= 1 2 ‖dk‖ may be suitable if the condition F (x∗k)− F (x ∗ k+dk) < 0.1 [Qk(x ∗ k)−Qk(x ∗ k+dk) ] (1.4) holds, because then the iteration fails to achieve one tenth of the reduction in the objective function that is predicted by the quadratic model. On the other hand, if