Learning-Rate Annealing Methods for Deep Neural Networks

Deep neural networks (DNNs) have achieved great success in the last decades. DNN is optimized using stochastic gradient descent (SGD) with learning rate annealing that overtakes adaptive methods many tasks. However, there no common choice regarding scheduled-annealing for SGD. This paper aims to present empirical analysis of based on experimental results major data-sets image classification one key applications DNNs. Our experiment involves recent deep network models combination a variety methods. We also propose an combining sigmoid function warmup shown overtake both and other existing schedules accuracy most cases