The Statistical Recurrent Unit
نویسندگان
چکیده
Sophisticated gated recurrent neural network architectures like LSTMs and GRUs have been shown to be highly effective in a myriad of applications. We develop an un-gated unit, the statistical recurrent unit (SRU), that is able to learn long term dependencies in data by only keeping moving averages of statistics. The SRU’s architecture is simple, un-gated, and contains a comparable number of parameters to LSTMs; yet, SRUs perform favorably to more sophisticated LSTM and GRU alternatives, often outperforming one or both in various tasks. We show the efficacy of SRUs as compared to LSTMs and GRUs in an unbiased manner by optimizing respective architectures’ hyperparameters in a Bayesian optimization scheme for both synthetic and realworld tasks.
منابع مشابه
Assessment of the Relationship between Recurrent High-risk Pregnancy and Mothers’ Previous Experience of Having an Infant Admitted to a Neonatal Intensive Care Unit
Background & aim: High-risk pregnancies increase the risk of Intensive Care Unit (ICU) and Neonatal Intensive Care Unit (NICU) admission in mothers and their newborns. In this study, we aimed to identify the association between the recurrence of high-risk pregnancy and mothers’ previous experience of having an infant admitted to NICU. Methods:We performed a cohort, retrospective study to compa...
متن کاملMinimum Translation Modeling with Recurrent Neural Networks
We introduce recurrent neural networkbased Minimum Translation Unit (MTU) models which make predictions based on an unbounded history of previous bilingual contexts. Traditional back-off n-gram models suffer under the sparse nature of MTUs which makes estimation of highorder sequence models challenging. We tackle the sparsity problem by modeling MTUs both as bags-of-words and as a sequence of i...
متن کاملTraumatic First Time Shoulder Dislocation: Surgery vs Non-Operative Treatment
Management of first shoulder dislocation following reduction remains controversial. The two main options are immobilisation and arthroscopic stabilisation. The aim of this article is to highlight some of the issues that influence decision making when discussing management options with these patients, including natural history of the first time dislocation, outcomes of surgery and non-operati...
متن کاملUnit Selection with Hierarchical Cascaded Long Short Term Memory Bidirectional Recurrent Neural Nets
Bidirectional recurrent neural nets have demonstrated state-ofthe-art performance for parametric speech synthesis. In this paper, we introduce a top-down application of recurrent neural net models to unit-selection synthesis. A hierarchical cascaded network graph predicts context phone duration, speech unit encoding and frame-level logF0 information that serves as targets for the search of unit...
متن کاملA New Recurrent Fuzzy Neural Network Controller Design for Speed and Exhaust Temperature of a Gas Turbine Power Plant
In this paper, a recurrent fuzzy-neural network (RFNN) controller with neural network identifier in direct control model is designed to control the speed and exhaust temperature of the gas turbine in a combined cycle power plant. Since the turbine operation in combined cycle unit is considered, speed and exhaust temperature of the gas turbine should be simultaneously controlled by fuel command ...
متن کامل