Stochastic quasi-Newton molecular simulations.

Stochastic Newton and Quasi-Newton Methods for Large Linear Least-squares Problems

We describe stochastic Newton and stochastic quasi-Newton approaches to efficiently solve large linear least-squares problems where the very large data sets present a significant computational burden (e.g., the size may exceed computer memory or data are collected in real-time). In our proposed framework, stochasticity is introduced in two different frameworks as a means to overcome these compu...

IQN: An Incremental Quasi-Newton Method with Local Superlinear Convergence Rate

The problem of minimizing an objective that can be written as the sum of a set of n smooth and strongly convex functions is challenging because the cost of evaluating the function and its derivatives is proportional to the number of elements in the sum. The Incremental Quasi-Newton (IQN) method proposed here belongs to the family of stochastic and incremental methods that have a cost per iterat...

A Variance Reduced Stochastic Newton Method

Quasi-Newton methods are widely used in practise for convex loss minimization problems. These methods exhibit good empirical performance on a wide variety of tasks and enjoy super-linear convergence to the optimal solution. For largescale learning problems, stochastic Quasi-Newton methods have been recently proposed. However, these typically only achieve sub-linear convergence rates and have no...

Supplementary Material for Stochastic Adaptive Quasi-Newton Methods for Minimizing Expected Values

Trust Regions and Ridge Analysis

Quasi-Newton methods for numerical optimization exploit quadratic Taylor polynomial models of the objective function. Trust regions are widely used to ensure the global convergence of these methods. Analogously, response surface methods for stochastic optimization exploit linear and quadratic regression models of the objective function. Ridge analysis is widely used to safeguard the optimizatio...

A Stochastic Quasi-Newton Method for Online Convex Optimization

We develop stochastic variants of the wellknown BFGS quasi-Newton optimization method, in both full and memory-limited (LBFGS) forms, for online optimization of convex functions. The resulting algorithm performs comparably to a well-tuned natural gradient descent but is scalable to very high-dimensional problems. On standard benchmarks in natural language processing, it asymptotically outperfor...