Fast Approximation of the Gauss--Newton Hessian Matrix for the Multilayer Perceptron

We introduce a fast algorithm for entrywise evaluation of the Gauss--Newton Hessian (GNH) matrix fully connected feed-forward neural network. The has precomputation step and sampling step. While it generally requires $\mathcal{O}(Nn)$ work to compute an entry (and entire column) in GNH network with $N$ parameters $n$ data points, our reduces cost $\mathcal{O}(n+d/\epsilon^2)$ work, where $d$ is output dimension $\epsilon$ prescribed accuracy (independent $N$). One application constructing hierarchical-matrix ($\mathcal{H}$-matrix) approximation solving linear systems eigenvalue problems. It $\mathcal{O}(N^2)$ memory $\mathcal{O}(N^3)$ store factorize matrix, respectively. $\mathcal{H}$-matrix only $\mathcal{O}(N r_o)$ footprint r_o^2)$ be factorized, $r_o \ll N$ maximum rank off-diagonal blocks matrix. demonstrate performance on classification autoencoder networks. (A corrected version attached.)