Methods for Encoding in Multilayer Feed-Forward Neural Networks

Neural network techniques for encodlng-decoding processes have been developed. The net we have devised can work like it memory retrieval system in the sense of Hopfield, Feinstein and Palmex. Its behaviour for 2 R (R E N) input units has some special interesting features. In particular, the accsssibilities for each initial symbol may be explicitly computed. Although thermal noise may muddle the code, we show how it can statistically rid the result of unwanted sequences while maintaining the network accuracy within a given bound. I n t r o d u c t i o n The idea of using layered neural networks for multiple tasks has become more and more appealing since Rosenblatt's original perceptron model was object of the first serious criticism which led to far-reaching developments [1]. Among this type of structures, the most interesting group are the multilayer feed-forward nets containing intermediate --or hidden-layers. The working of any net is determined by the relative strength of the links among units (neurons), usually given by a weight or connection exchange matrix. Typica~y, in a feed-forward multilayer network each unit computes a nonlinear function of the weighted sum of incoming signals from the previous layer reaching its own site, and sends the outcome on to the following layer. This process ends when the emerging signal arrives at the output units, where the result is read off. Mdltilayer feed-forward neural networks are specially adequate for encoding [2], understood as the turning of p possible input patterns described by N digital units into a determined set of p output patterns on M units. In the minimal set-up, there is just one hidden layer of R hidden units forming a binary representation of the N inputs, i.e. with /~ = log2N neurons (Fig. 1). We will take M = N = p in order to have the same number of neurons at the input and output layers. The first case to be considered is that in which we have sets ---or alphabets-of unary patterns, i.e. binary sequences in which one unit is on and the rest are off: 1 ~ 1 ~ ~+1 N ~ " = ( , . . . . , + , . . . . ) , ~ = 1 . . . . . N. (1) Afterwards, we shall consider arbitrary input and output patterns. When the number of output units has the form 2 R, with R E N, our model exhibits remarkable characteristics. In particula L no thresholds will be needed in this case. A no-go theorem on the possibility of general 3-step encoding will be proven. At the end, advantage will be taken of the introduction of a moderate amount of thermal noise. For the intermediate and output layers, the state of each unit at a given moment will be a (generally nonlinear) function of the weighted sum of the signals feeding into it. Since we use binary units, we take the