An efficient algorithm for recurrent neural network training is presented. The approach increases the training speed for tasks where a length of the input sequence may vary significantly. The proposed approach is based on the optimal batch bucketing by input sequence length and data parallelization on multiple graphical processing units. The baseline training performance without sequence bucket...

Recurrent networks are trained to memorize their input better, often in the hopes that such training will increase the ability of the network to predict. We show that networks designed to memorize input can be arbitrarily bad at prediction. We also find, for several types of inputs, that one-node networks optimized for prediction are nearly at upper bounds on predictive capacity given by Wiener...

We explore the dynamics of a Hodgkin-Huxley-type model for thermally sensitive neurons that exhibit intrinsic oscillatory activity. The model is modified to include a feedback loop that is represented by two parameters: the synaptic strength and the transmission delay time. We analyze the dynamics of the neuron depending on the temperature, the synaptic strength, and the delay time. We find par...

We put forward a model for neural representation of intervals of time. The model is comprised of ordinary recurrent neural networks. Assumptions speci"c to our model are the following two: membrane potential of each neuron is bistable; each neuron receives random noise input in addition to the recurrent input. Results of computer simulation show that the network activity triggered at an initial...

We replace the Hidden Markov Model (HMM) which is traditionally used in in continuous speech recognition with a bi-directional recurrent neural network encoder coupled to a recurrent neural network decoder that directly emits a stream of phonemes. The alignment between the input and output sequences is established using an attention mechanism: the decoder emits each symbol based on a context cr...

This paper presents a novel approach using recurrent neural networks for estimating the quality of machine translation output. A sequence of vectors made by the prediction method is used as the input of the final recurrent neural network. The prediction method uses bi-directional recurrent neural network architecture both on source and target sentence to fully utilize the bi-directional quality...

A recurrent neural network is applied for minimizing the infinity-norm of joint torques in redundant manipulators. The recurrent neural network explicitly minimizes the maximum component of joint torques in magnitude while keeping the relation between the joint torque and the end-effector acceleration satisfied. The end-effector accelerations are given to the recurrent neural network as its inp...

We present the discriminative recurrent sparse auto-encoder model, comprising a recurrent encoder of rectified linear units, unrolled for a fixed number of iterations, and connected to two linear decoders that reconstruct the input and predict its supervised classification. Training via backpropagation-through-time initially minimizes an unsupervised sparse reconstruction error; the loss functi...