Exploration of combining ESN learning with gradient-descent RNN learning techniques

Modeling dynamical systems via neural networks has become a well-established field of research in Computer Science. However, for more complex dynamical systems the current algorithms are often impractical or not accurate enough. Recurrent Neural Networks (RNNs), when coupled with the Echo-State Network (ESN) approach, offer an efficient way of solving such kind of problems – by concentrating on the behavior that can be observed from the outside and modeling it, while ignoring the underlying processes. This biologically inspired method produces very good results for a variety of problems and, in many cases, out-beats other algorithms' results by orders of magnitude. However, for certain scenarios, the ESN algorithm requires large networks in order to operate at its full potential. This is not desirable in those situations where the physical or computational requirements limit the size of network to be used (such as the case of micro-controllers and embedded devices). The aim of this project was to provide a remedy to this problem by employing a " post-processing " technique on the solutions generated by the ESN algorithm. This technique takes the form of a traditional algorithm for learning with RNNs. While these algorithms are normally slow and tend to produce worse results, they require small networks in order to operate properly. This way we found a compromise solution that helped us show that it is possible to employ the ESN algorithm even on small networks and still obtain good results by means of " post-processing " them with other, more traditional, algorithms.