Many iterative methods that are used to solve Ax = b can be derived as quasi-Newton methods for minimizing the quadratic function 1 2 xAAx−xAb. In this paper, several such methods are considered, including conjugate gradient least squares (CGLS), Barzilai-Borwein (BB), residual norm steepest descent (RNSD) and Landweber (LW). Regularization properties of these methods are studied by analyzing t...

{ We describe a Newton method applied to the evaluation of a critical point of a total energy associated to a shape optimization problem. The key point of these methods is the Hessian of the shape functional. We give an expression of the Hessian as well as the relation with the second-order Eulerian semi-derivative. An application to the electromagnetic shaping of liquid metals process is studi...

In this paper we analyze the use of structured quasi-Newton formulae as preconditioners of iterative linear methods when the inexact-Newton approach is employed for solving nonlinear systems of equations. We prove that superlinear convergence and bounded work per iteration is obtained if the preconditioners satisfy a Dennis-Moré condition. We develop a theory of LeastChange Secant Update precon...

We derive an asymptotic Newton algorithm for Quasi Maximum Likelihood estimation of the ICA mixture model, using the ordinary gradient and Hessian. The probabilistic mixture framework can accommodate non-stationary environments and arbitrary source densities. We prove asymptotic stability when the source models match the true sources. An application to EEG segmentation is given. Index Terms Ind...

This paper reviews advances in Newton quasi Newton and conjugate gradi ent methods for large scale optimization It also describes several packages developed during the last ten years and illustrates their performance on some practical problems Much attention is given to the concept of partial separa bility which is gaining importance with the arrival of automatic di erentiation tools and of opt...

Second order methods for optimization call for the solution of sequences of linear systems. In this survey we will discuss several issues related to the preconditioning of such sequences. Covered topics include both techniques for building updates of factorized preconditioners and quasi-Newton approaches. Sequences of unsymmetric linear systems arising in NewtonKrylov methods will be considered...

Presented here is a generalization of the modified relative Newton method, recently proposed in [1] for quasi-maximum likelihood blind source separation. Special structure of the Hessian matrix allows to perform block-coordinate Newton descent, which significantly reduces the algorithm computational complexity and boosts its performance. Simulations based on artificial and real data show that t...

Limited memory quasi-Newton methods and trust-region methods represent two efficient approaches used for solving unconstrained optimization problems. A straightforward combination of them deteriorates the efficiency of the former approach, especially in the case of large-scale problems. For this reason, the limited memory methods are usually combined with a line search. We show how to efficient...

We study the numerical performance of a limited memory quasi Newton method for large scale optimization which we call the L BFGS method We compare its performance with that of the method developed by Buckley and LeNir which combines cyles of BFGS steps and conjugate direction steps Our numerical tests indicate that the L BFGS method is faster than the method of Buckley and LeNir and is better a...

In this paper, we present a generic framework to extend existing uniformly optimal convex programming algorithms to solve more general nonlinear, possibly nonconvex, optimization problems. The basic idea is to incorporate a local search step (gradient descent or Quasi-Newton iteration) into these uniformly optimal convex programming methods, and then enforce a monotone decreasing property of th...