Improved Preconditioner for Hessian Free Optimization

نویسنده

  • Olivier Chapelle
چکیده

We investigate the use of Hessian Free optimization for learning deep autoencoders. One of the critical components in that algorithm is the choice of the preconditioner. We argue in this paper that the Jacobi preconditioner leads to faster optimization and we show how it can be accurately and efficiently estimated using a randomized algorithm.

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Preconditioning for Hessian-Free Optimization

Recently Martens adapted the Hessian-free optimization method for the training of deep neural networks. One key aspect of this approach is that the Hessian is never computed explicitly, instead the Conjugate Gradient(CG) Algorithm is used to compute the new search direction by applying only matrix-vector products of the Hessian with arbitrary vectors. This can be done efficiently using a varian...

متن کامل

Automatic Preconditioning by Limited Memory Quasi-Newton Updating

This paper proposes a preconditioner for the conjugate gradient method (CG) that is designed for solving systems of equations Ax = bi with different right-hand-side vectors or for solving a sequence of slowly varying systems Akx = bk. The preconditioner has the form of a limited memory quasi-Newton matrix and is generated using information from the CG iteration. The automatic preconditioner doe...

متن کامل

Preconditioning the Pressure Tracking in Fluid Dynamics by Shape Hessian Information

Potential flow pressure matching is a classical inverse design aerodynamic problem. The resulting loss of regularity during the optimization poses challenges for shape optimization with normal perturbation of the surface mesh nodes. Smoothness is not enforced by the parameterization but by a proper choice of the scalar product based on the shape Hessian, which is derived in local coordinates fo...

متن کامل

Automatic Preconditioning by Limited Memory

The paper proposes a preconditioner for the conjugate gradient method CG that is designed for solving systems of equations Ax bi with di erent right hand side vec tors or for solving a sequence of slowly varying systems Akx bk The preconditioner has the form of a limited memory quasi Newton matrix and is generated using infor mation from the CG iteration The automatic preconditioner does not re...

متن کامل

Impulse Response Approximations of Discrete Shape Hessians with Application in CFD

Abstract. This paper discusses the symbol of the Hessian of a shape optimization problem in a viscid, incompressible flow. The symbol of the Hessian for the Stokes shape optimization problem is analytically approximated by tracking a Fourier mode through the operators involved. We propose a discrete method for finding the symbol and confirm the correctness by comparison with the analytical data...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

عنوان ژورنال:

دوره   شماره 

صفحات  -

تاریخ انتشار 2011