Optimization in Deep Learning is mainly guided by vague intuitions and strong assumptions, with a limited understanding how why these work practice. To shed more light on this, our provides some deeper understandings of SGD behaves empirically analyzing the trajectory taken from line search perspective. Specifically, costly quantitative analysis full-batch loss along trajectories common used mo...

Related DatabasesWeb of Science You must be logged in with an active subscription to view this.Article DataHistorySubmitted: 13 May 2020Accepted: 02 March 2021Published online: 20 2021Keywordslow-rank matrices, optimization on manifolds, multilevel optimization, Riemannian retraction-based line search, roundoff errorAMS Subject Headings65F10, 65N22, 65F50, 65K10Publication DataISSN (print): 106...

In this paper‎, ‎two extended three-term conjugate gradient methods based on the Liu-Storey ({tt LS})‎ ‎conjugate gradient method are presented to solve unconstrained optimization problems‎. ‎A remarkable property of the proposed methods is that the search direction always satisfies‎ ‎the sufficient descent condition independent of line search method‎, ‎based on eigenvalue analysis‎. ‎The globa...