Information Theory of a Multilayer Neural Network with Discrete Weights

Statistical mechanics is applied to estimate the maximal capacity per weight (a,) of a two-layer feed-forward network with discrete weights of depth 1, functioning as a parity machine of the K hidden units. For each K and lSZo(K'), the maximal theoretical capacity a, = 10g2(2Z) is achieved, the capacity per bit is 1, the average overlap between different solutions is zero and Zo(K) logK for large K. At finite temperature, a one-step replica symmetry-breaking solution is found to be exact for I < Zo(K). In the recent past, statistical-mechanical methods have been applied to investigate the properties of neural networks. Among these systems, the class of multilayer networks plays an important role[l]. The prototype of this class of architecture is the one-layer perceptron [l], consisting of one input layer of N binary units and one-binary output unit. Various statistical mechanical properties of the one-layer perceptron such as the maximal capacity and the generalization ability of the network have been recently investigated by using the pioneering work of Elizabeth Gardner [2,3]. However, the computational capability of a one-layer perceptron is limited, since it cannot solve nonseparable problems. Furthermore, multilayer networks may play an important role in the functioning of biological systems. In particular, it is important to examine the advantages of multilayer networks over the perceptron with respect to quantities such as storage capacity per weight and the effect of synaptic depth (the resolution of the synaptic strength). Furthermore, the applicability of neural networks to biology and to the construction of real devices requires the understanding of the interplay between the synaptic depth and the properties of the network. The effect of the synaptic depth on the system is crucial, since the implementation of a deeper synaptic depth is much more difficult and expensive in real neural network circuits [4]. Hence, an optimal depth should be defined and predicted for each task of the network. In this letter a two-layer feed-forward network is studied using a statistical-mechanical approach. The architecture of the network consists of N binary input units, one hidden layer with K continuous or discrete units and a single-binary output unit. The input units are divided into K equal disjoint sets, each one consisting of NIK units. The j ' s hidden unit is connected only to the i-th input via a weight of depth 1, J , = +1, +2, ..., k I, such that N ( j 1)/K < i < Nj/K. The configuration of the input is denoted by {st} , i = 1, .. . , N, with 182 EUROPHYSICS LETTERS si = k 1. The state of the j-th hidden unit is equal to its induced local field