Low-Dimensional Manifolds Support Multiplexed Integrations in Recurrent Neural Networks

We study the learning dynamics and representations emerging in recurrent neural networks (RNNs) trained to integrate one or multiple temporal signals. Combining analytical numerical investigations, we characterize conditions under which an RNN with n neurons learns D(?n) scalar signals of arbitrary duration. show, for linear, ReLU, sigmoidal neurons, that internal state lives close a D-dimensional manifold, whose shape is related activation function. Each neuron therefore carries, various degrees, information about value all integrals. discuss deep analogy between our results concept mixed selectivity forged by computational neuroscientists interpret cortical recordings.