Training Recurrent Neural Networks via Trajectory Modification

Traj ectory modification of recurrent neur al networks is a training algorithm that modifies both th e network representat ions in each tim e step and th e common weight matrix. The present algorithm is a genera lization of th e energy minimization formalism for tr ainin g feed-forward networks via modifications of th e int ern al represent ati ons. In a previous paper we showed that th e same form alism leads to th e back-propagation algorithm for cont inuous neurons and to a generalization of the CHIR training procedure for binary neur ons. The TRAM algorithm adopts a similar approach for tr aining recurrent neur al net works with stable endpoints , whereby the network representa tion s in each time step may be modified in parallel to the weight matrix. In carry ing out the analysis, consistency with other training algorithms is demonstrat ed when a cont inuous-valued system is considered, while th e TRAM learning procedure, repr esentin g an ent irely different concept, is obtained for the discrete case. Computer simulations carr ied out for th e restricted cases of parity and teacher-n et problems show rapid convergence of th e algorit hm.