Conjugate Gradient Methods in Training Neural Networks

Training of artificial neural networks is normally a time consuming task due to iterative search imposed by the implicit nonlinearity of the network behavior. To tackle the supervised learning of multilayer feed forward neural networks, the backpropagation algorithm has been proven to be one of the most successful neural network algorithm. Although backpropagation training has proved to be efficient in many applications, its convergence tends to be very slow and it often yields suboptimal solutions. Standard backpropagation, as with many gradient based optimizaton methods converges slowly as neural networks problems become larger and more complex. This paper concentrates on conjugate gradient-based training methods originated from optimization theory, namely, Fletcher Reeves conjugate gradient, Polak-Ribierre conjugate gradient and Powell-Beale restart. The behavior of these training methods on several real life application problems is reported, thereby illuminating convergence and robustness. The real world problems which have been considered include Classification of Iris Plant, Gender Classification of Crabs and Classification of Face Images. By using these algorithms, the convergence rate can be improved immensely with only a minimal increase in the complexity. Numerical evidence shows that these methods do perform well. (ATCMA264)