A method is provided for designing and training noise-driven recurrent neural networks as models of stochastic processes. The method unifies and generalizes two known separate modeling approaches, Echo State Networks (ESN) and Linear Inverse Modeling (LIM), under the common principle of relative entropy minimization. The power of the new method is demonstrated on a stochastic approximation of t...

Time series often have a temporal hierarchy, with information that is spread out over multiple time scales. Common recurrent neural networks, however, do not explicitly accommodate such a hierarchy, and most research on them has been focusing on training algorithms rather than on their basic architecture. In this paper we study the effect of a hierarchy of recurrent neural networks on processin...

Time series often have a temporal hierarchy, with information that is spread out over multiple time scales. Common recurrent neural networks, however, do not explicitly accommodate such a hierarchy, and most research on them has been focusing on training algorithms rather than on their basic architecture. In this paper we study the effect of a hierarchy of recurrent neural networks on processin...

Abs t rac t . Approaches to data mining proposed so far are mainly symbolic decision trees and numerical feedforward neural networks methods. While decision trees give, in many cases, lower accuracy compared to feedforward neural networks, the latter show black-box behaviour, long training times, and difficulty to incorporate available knowledge. We propose to use an incrementally-generated rec...

Recurrent neural networks have recently been demonstrated to have the ability to learn simple grammars. In particular, networks using second-order units have been successfully at this task. However, it is often difficult to predict the optimal neural network size to induce an unknown automaton from examples. Instead of just adjusting the weights in a network of fixed topology, we adopt the dyna...

We train recurrent neural network on a single, long, complex symbolic sequence with positive entropy. Training process is monitored through information theory based performance measures. We show that although the sequence is unpredictable, the network is able to code the sequence topological and statistical structure in recurrent neurons' activation scenarios. Such scenarios can be compactly re...

Neural and neuro-fuzzy models are powerful nonlinear modelling tools. Different structures, with different properties, are widely used to capture static or dynamical nonlinear mappings. Static (non-recurrent) models share a common structure: a nonlinear stage, followed by a linear mapping. In this paper, the separability of linear and nonlinear parameters is exploited for completely supervised ...