A high capacity incremental and local learning algorithm for attractor neural networks

Attractor neural networks such as the Hop eld network can be trained by a number of algorithms. The simplest of these methods is the Hebb rule which is strictly local (the weight of the synapse depends only on the activation of the two neurons it connects) and incremental (adding a new memory can be done knowing only the old weight matrix and not the actual patterns previously stored). Both of these properties are highly desirable. However it has been shown that the Hebbian network has a low absolute capacity of n=(2 ln n) where n is the total number of neurons. This capacity can be increased to n by use of the pseudo-inverse rule. However this is neither local nor incremental. At best the pseudo-inverse weight matrix can be generated by local (not strictly local) and non-incremental limit processes. The question addressed by this paper is, 'Can the capacity of the Hebbian rule be increased without losing locality or incrementality?' Here a new algorithm is proposed. This algorithm is local but not strictly local: The weight depends on both the activation and local elds of the two neurons that it connects. It is immediate: The learning process is one shot rather than a limit process. It is also incremental. In addition it has an absolute capacity signi cantly higher than that of the Hebbian method: n=p2 lnn In this report the new learning method is introduced. The relationship between the new rule, the Hebbian rule and the pseudo-inverse is given, the absolute capacity of the learning algorithm is calculated rigourously, using a simple probabilistic approach. Complicated large deviation calculations are avoided by a judicious use of an exponential upper bound. Simulations are provided to show that this calculation does indeed provide a good measure of the capacity for nite network sizes. Chapter