Modified Kalman filter based method for training state-recurrent multilayer perceptrons
نویسندگان
چکیده
Kalman filter based training algorithms for recurrent neural networks provide a clever alternative to the standard backpropagation in time. However, these algorithms do not take into account the optimization of the hidden state variables of the recurrent network. In addition, their formulation requires Jacobian evaluations over the entire network, adding to their computational complexity. In this paper, we propose a spatial-temporal extended Kalman filter algorithm for training recurrent neural network weights and internal states. This new formulation also reduces the computational complexity of Jacobian evaluations drastically by decoupling the gradients of each layer. Monte Carlo comparisons with backpropagation through time point out the robust and fast convergence of the algorithm.
منابع مشابه
Direct Method for Training Feed-Forward Neural Networks Using Batch Extended Kalman Filter for Multi-Step-Ahead Predictions
This paper is dedicated to the long-term, or multi-step-ahead, time series prediction problem. We propose a novel method for training feed-forward neural networks, such as multilayer perceptrons, with tapped delay lines. Special batch calculation of derivatives called Forecasted Propagation Through Time and batch modification of the Extended Kalman Filter are introduced. Experiments were carrie...
متن کاملRecurrent Multilayer Perceptrons for Identiication and Control: the Road to Applications
This study investigates the properties of artiicial recurrent neural networks. Particular attention is paid to the question of how these nets can be applied to the identiication and control of non-linear dynamic processes. Since these kind of processes can only insuuciently be modelled by conventional methods, diierent approaches are required. Neural networks are considered to be useful for thi...
متن کاملRecurrent Multilayer Perceptrons for Identi cation and Control : The Road to Applications
This study investigates the properties of artiicial recurrent neural networks. Particular attention is paid to the question of how these nets can be applied to the identiication and control of non-linear dynamic processes. Since these kind of processes can only insuuciently bemodelled by conventional methods, diierent approaches are required. Neural networks are considered to be useful for this...
متن کاملDevelopment of A New Recurrent Neural Network Toolbox (RNN-Tool)
In this report, we developed a new recurrent neural network toolbox, including the recurrent multilayer perceptron structure and its companying extended Kalman filter based training algorithms: BPTT-GEKF and BPTT-DEKF. Besides, we also constructed programs for designing echo state network with single reservoir, together with the offline linear regression based training algorithm. We name this t...
متن کاملPrediction of Dynamical Systems by Recurrent Neural Networks
Recurrent neural networks in general achieve better results in prediction of time series then feedforward networks. Echo state neural networks seem to be one alternative to them. I have shown on the task of text correction, that they achieve slightly better results compared to already known method based on Markov model. The major part of this work is focused on alternatives to recurrent neural ...
متن کامل