Information theoretic limits of learning a sparse rule

Abstract We consider generalized linear models in regimes where the number of nonzero components signal and accessible data points are sublinear with respect to size signal. prove a variational formula for asymptotic mutual information per sample when system grows infinity. This result allows us derive an expression minimum mean-square error (MMSE) Bayesian estimator entries have discrete distribution finite support. find that, such signals suitable vanishing scalings sparsity sampling rate, MMSE is nonincreasing piecewise constant. In specific instances even displays all-or-nothing phase transition, that is, sharply jumps from its maximum value zero at critical rate. The phenomenon has previously been shown occur high-dimensional regression. Our analysis goes beyond case applies learning weights perceptron general activation function teacher-student scenario. particular, we discuss generalization set training examples.