On Centering Neural Network Weight Updates ?
نویسنده
چکیده
It has long been known that neural networks can learn faster when their input and hidden unit activities are centered about zero; recently we have extended this approach to also encompass the centering of error signals (Schraudolph and Sejnowski, 1996). Here we generalize this notion to all factors involved in the weight update, leading us to propose centering the slope of hidden unit activation functions as well. Slope centering removes the linear component of backpropagated error; this improves credit assignment in networks with shortcut connections. Benchmark results show that this can speed up learning signiicantly without adversely aaecting the trained network's generalization ability.
منابع مشابه
Improving Accuracy of DGPS Correction Prediction in Position Domain using Radial Basis Function Neural Network Trained by PSO Algorithm
Differential Global Positioning System (DGPS) provides differential corrections for a GPS receiver in order to improve the navigation solution accuracy. DGPS position signals are accurate, but very slow updates. Improving DGPS corrections prediction accuracy has received considerable attention in past decades. In this research work, the Neural Network (NN) based on the Gaussian Radial Basis Fun...
متن کاملIDSIA - 19 - 97 April 19 , 1997 revised August 21 , 1998 Centering Neural Network Gradient Factors ?
It has long been known that neural networks can learn faster when their input and hidden unit activities are centered about zero; recently we have extended this approach to also encompass the centering of error signals [2]. Here we generalize this notion to all factors involved in the network’s gradient, leading us to propose centering the slope of hidden unit activation functions as well. Slop...
متن کاملHow to Center Binary Deep Boltzmann Machines
This work analyzes centered binary Restricted Boltzmann Machines (RBMs) and binary Deep Boltzmann Machines (DBMs), where centering is done by subtracting offset values from visible and hidden variables. We show analytically that (i) centering results in a different but equivalent parameterization for artificial neural networks in general, (ii) the expected performance of centered binary RBMs/DB...
متن کاملTense, Mood, and Centering
I propose that tense and mood paradigms are grammatical centering systems. Specifically, English tenses form a temporal centering system, which monitors and updates topic times, whereas Kalaallisut moods form a modal centering system, which monitors and updates modal discourse referents. Nevertheless, English and Kalaallisut translation equivalents converge on the same truth conditions, due to ...
متن کاملCentering Neural Network Gradient Factors
It has long been known that neural networks can learn faster when their input and hidden unit activities are centered about zero; recently we have extended this approach to also encompass the centering of error signals 2]. Here we generalize this notion to all factors involved in the network's gradient, leading us to propose centering the slope of hidden unit activation functions as well. Slope...
متن کامل