Minimisation methods for training feedforward neural networks

-Mimmtsatlon methods for trammgfeedforward networks with back propagatton are compared Feedforward neural network trammg ts a special case of functlon mmtmtsatton, where no exphctt model o f the data ts assumed Therefore, and due to the htgh dlmenstonahty o f the data, hneartsatton of the trainmg problem through use o f orthogonal basts functtons is not destrable The focus ts on functton mmlmtsatton on any basts Quast-Newton and conJugate gradtent methods are revtewed, and the latter are shown to be a spectal case o f error back propagatton wtth momentum term Three feedforward learnmg problems are tested wtth five methods It ts shown that, due to the f ixed stepstze, standard error back propagatton performs well In avotdlng local mtmma. However, by usmg not only the local gradtent but also the second dertvattve o f the error function, a much shorter trammg ttme ts requtred ConJugate gradtent with Powell restarts shows to be the supertor method Keywords--Feedforward neural network training, Numerical optlmisatlon techniques, Neural funcUon approximation, Error back propagation, Conjugate gradient, Quas~-Newton.