It has been established that the second-order stochastic gradient descent (2SGD) method can potentially achieve generalization performance as well as empirical optimum in a single pass (i.e., epoch) through the training examples. However, 2SGD requires computing the inverse of the Hessian matrix of the loss function, which is prohibitively expensive. This paper presents Periodic Step-size Adapt...

This paper presents a new, fast algorithm for finite-length minimum mean square error (MMSE) equalizers. The research exploits asymptotic equivalence of Toeplitz and circulant matrices to estimate Hessian matrix of a quadratic form. Research shows that the Hessian matrix exhibits a specific structure. As a result, when combined with the Rayleigh minimization algorithm, it provides an efficient ...

We present a method for approximately inverting the Hessian of full waveform inversion as a dip-dependent and scaledependent amplitude correction. The terms in the expansion of this correction are determined by least-squares fitting from a handful of applications of the Hessian to random models — a procedure called matrix probing. We show numerical indications that randomness is important for g...

We discuss the role of automatic differentiation tools in optimization software. We emphasize issues that are important to large-scale optimization and that have proved useful in the installation of nonlinear solvers in the NEOS Server. Our discussion centers on the computation of the gradient and Hessian matrix for partially separable functions and shows that the gradient and Hessian matrix ca...

A Hessian matrix in full waveform inversion (FWI) is difficult to compute directly because of high computational cost and an especially large memory requirement. Therefore, Newton-like methods are rarely feasible in realistic large-size FWI problems. We modify the quasi-Newton BFGS method to use a projected Hessian matrix that reduces both the computational cost and memory required, thereby mak...

In this technical report we derive the analytic form of the Hessian matrix for shape matching energy. Shape matching (Fig. 1) is a useful technique for meshless deformation, which can be easily combined with multiple techniques in real-time dynamics (refer to [MHTG05, BMM15] for more details). Nevertheless, it has been rarely applied in scenarios where implicit (such as backward differentiation...

The k-Hessian is the k-trace, or the kth elementary symmetric polynomial of eigenvalues of the Hessian matrix. When k ≥ 2, the k-Hessian equation is a fully nonlinear partial differential equations. It is elliptic when restricted to k-admissible functions. In this paper we establish the existence and regularity of k-admissible solutions to the Dirichlet problem of the k-Hessian equation. By a g...

We present a new Hessian estimator based on the simultaneous perturbation procedure, that requires three system simulations regardless of the parameter dimension. We then present two Newton-based simulation optimization algorithms that incorporate this Hessian estimator. The two algorithms differ primarily in the manner in which the Hessian estimate is used. Both our algorithms do not compute t...

Modified Shepard interpolation based on second order Taylor series expansions has proven to be a flexible tool for constructing potential energy surfaces in a range of situations. Extending this to gas-surface dynamics where surface atoms are allowed to move represents a substantial increase in the dimensionality of the problem, reflected in a dramatic increase in the computational cost of the ...

Several well known integral stochastic orders (like the convex order, the supermodular order, etc.) can be defined in terms of the Hessian matrix of a class of functions. Here we consider a generic Hessian order, i.e., an integral stochastic order defined through a convex coneH of Hessian matrices, and we prove that if two random vectors are ordered by the Hessian order, then their means are eq...