We propose a generalization of the conjugate gradient method that uses multiple preconditioners, combining them automatically in an optimal way. The derivation is described in detail, and analytical observations are made. A short recurrence relation does not hold in general for this new method, but in at least one case such a relation is satisfied: for two symmetric positive definite preconditi...

Natural gradient is a gradient descent technique which uses the inverse of the covariance matrix of the gradient. Using the centrallimit theorem, we prove that it yields the direction that minimizes the probability of overfitting. However, its prohibitive computational cost makes it impractical for online training. Here, we present a new online version of the natural gradient which we coin TONG...

This paper considers the least-square online gradient descent algorithm in a reproducing kernel Hilbert space (RKHS) without an explicit regularization term. We present a novel capacity independent approach to derive error bounds and convergence results for this algorithm. The essential element in our analysis is the interplay between the generalization error and a weighted cumulative error whi...

In a previous report [3], a methodology for the numerical treatment of a two-objective optimization problem, possibly subject to equality constraints, was proposed. The method was devised to be adapted to cases where an initial design-point is known and such that one of the two disciplines, considered to be preponderant, or fragile, and said to be the primary discipline, achieves a local or glo...

We consider the problem of minimizing a convex objective which is the sum of a smooth part, with Lipschitz continuous gradient, and a nonsmooth part. Inspired by various applications, we focus on the case when the nonsmooth part is a composition of a proper closed convex function P and a nonzero affine map, with the proximal mappings of τP , τ > 0, easy to compute. In this case, a direct applic...

We present a new gradient algorithm to perform blind signal separation (BSS). The algorithm is obtained by taking a trade-off between the ordinary gradient algorithm and the natural gradient algorithm. It provides a better performance than the ordinary gradient algorithm and is free from small-step-size restriction of the natural gradient algorithm. In addition, the algorithm has less computati...

The Polyhedral Active Set Algorithm (PASA) is designed to optimize a general nonlinear function over polyhedron. Phase one of the algorithm nonmonotone gradient projection algorithm, while phase two an active set that explores faces constraint A gradient-based implementation presented, where projected version conjugate employed in two. Asymptotically, only performed. Comparisons are given with ...

in this paper, the artificial neural network (ann) approach is applied for forecasting groundwater level fluctuation in aghili plain,southwest iran. an optimal design is completed for the two hidden layers with four different algorithms: gradient descent withmomentum (gdm), levenberg marquardt (lm), resilient back propagation (rp), and scaled conjugate gradient (scg). rain,evaporation, relative...

in this paper, the artificial neural network (ann) approach is applied for forecasting groundwater level fluctuation in aghili plain,southwest iran. an optimal design is completed for the two hidden layers with four different algorithms: gradient descent withmomentum (gdm), levenberg marquardt (lm), resilient back propagation (rp), and scaled conjugate gradient (scg). rain,evaporation, relative...

The stability of learning rate in neural network identifiers and controllers is one of the challenging issues which attracts great interest from researchers of neural networks. This paper suggests adaptive gradient descent algorithm with stable learning laws for modified dynamic neural network (MDNN) and studies the stability of this algorithm. Also, stable learning algorithm for parameters of ...