In this paper, we propose a second order optimization method to learn models where both the dimensionality of the parameter space and the number of training samples is high. In our method, we construct on each iteration a Krylov subspace formed by the gradient and an approximation to the Hessian matrix, and then use a subset of the training data samples to optimize over this subspace. As with t...

Newton-type methods for unconstrained optimization problems have been very successful when coupled with a modified Cholesky factorization to take into account the possible lack of positivedefiniteness in the Hessian matrix. In this paper we discuss the application of these methods to large problems that have a sparse Hessian matrix whose sparsity is known a priori. Quite often it is difficult, ...

In this paper, we study the 66 Cartesian stiiness matrix. We show that the stiiness of a rigid body subjected to conservative forces and moments is described by a (0; 2) tensor which is the Hessian of the potential function. The key observation of the paper is that since the Hessian depends on the choice of an aane connection in the task space, so will the Carte-sian stiiness matrix. Further, t...

L-convex functions are nonlinear discrete functions on integer points that are computationally tractable in optimization. In this paper, a discrete Hessian matrix and a local quadratic expansion are defined for L-convex functions. We characterize L-convex functions in terms of the discrete Hessian matrix and the local quadratic expansion.

Introduction Vessel enhancement filters applied to 3D MRA data sets prior to rendering as a 2D image may improve visualization of vessel detail. We previously compared several line enhancement filters for intracranial 3D MR angiography images[1]. We examined filters based on discrete lines (e.g., Du and Parker's filter [2]) and on the Hessian matrix (Frangi [3]). We found the Du and Parker filt...

A Hessian matrix in full waveform inversion (FWI) is difficult to compute directly because of high computational cost and an especially large memory requirement. Therefore, Newton-like methods are rarely feasible in realistic largesize FWI problems. We modify the BFGS method to use a projected Hessian matrix that reduces both the computational cost and memory required, thereby making a quasiNew...

In recent years the Hessian matrix and its eigenvalues became important in pattern recognition. Several algorithms based on the information they provide have been introduced. We recall the relationship between the eigenvalues of Hessian matrix and the 2nd order edge detection filter, show the usefulness of treating them separately and exploit these facts to design a combined threshold operation...

In this paper we focus on the linear algebra theory behind feedforward (FNN) and recurrent (RNN) neural networks. We review backward propagation, including backward propagation through time (BPTT). Also, we obtain a new exact expression for Hessian, which represents second order effects. We show that for t time steps the weight gradient can be expressed as a rank-t matrix, while the weight Hess...