Learning rate selection in stochastic gradient methods based on line search strategies

Finite-sum problems appear as the sample average approximation of a stochastic optimization problem and often arise in machine learning applications with large scale data sets. A very popular approach to face finite-sum is gradient method. It well known that proper strategy select hyperparameters this method (i.e. set a-priori selected parameters) and, particular, rate, needed guarantee convergence properties good practical performance. In paper, we analyse standard line search based updating rules fix rate sequence, also relation size mini batch chosen compute current gradient. An extensive numerical experimentation carried out order evaluate effectiveness discussed strategies for convex non-convex test problems, highlighting methods avoid expensive initial setting hyperparameters. The approaches have been applied train Convolutional Neural Network, providing promising results.