Noisy Linear Networks

During the last few years, (bottleneck) linear neural network architectures have received considerable amounts of interest due to their importance in feature extraction. Suppose that an n-dimensional random vector x is to be compressed into a p-dimensional vector y = c(x), where p < n, in a way that relative to some performance criterion, y contains as much information about x as possible. If the mean square error of the best linear estimate of x from y (the \linear reconstruction error") is used as criterion and x is centered and square integrable, extraction of the rst p principal components of the input covari-ance matrix = Cov(x), a linear compression method, is optimal (Bourlard & Kamp, 1988). I.e., the error function I Ejx?Bc(x)j 2 is minimized for B = U p T ?1 and c(x) = TU 0 p x, where the p columns of U p are mutually perpendicular unit length eigenvectors of associated with the p largest eigenvalues of , and T is an invertible p p matrix. If in addition the data is gaussian, these choices also minimize the mutual information between x and y, cf. e.g. Baldi & Hornik (1992). In this paper, we are concerned with situations where a linear feature extraction process is contaminated by noise. More precisely, suppose that upon presentation of an input x, the network actually computes y = Ax + e, where e is the processing noise. Such noise might e.g. be caused by intrinsic unreliability of the network units or external noise during transmission of the signals. We assume that e is centered with covariance matrix Cov(e) = R and uncorrelated with x. Here, is the noise level, and R describes the noise structure (if e.g. the output units are physically close, it may be unrealistic to assume that the noise components are uncorrelated). Clearly, this is not the only possible noise model. For ease of exposition, we also assume that both and R are strictly positive. In this case, the optimal linear reconstruction By of x from y = Ax + e is obtained with B = B (A) = A 0 (AA 0 +R) + , and the conditional covariance