Scrambled Objects for Least-Squares Regression

We consider least-squares regression using a randomly generated subspace GP ⊂ F of finite dimension P , where F is a function space of infinite dimension, e.g. L2([0, 1]). GP is defined as the span of P random features that are linear combinations of the basis functions of F weighted by random Gaussian i.i.d. coefficients. In particular, we consider multi-resolution random combinations at all scales of a given mother function, such as a hat function or a wavelet. In this latter case, the resulting Gaussian objects are called scrambled wavelets and we show that they enable to approximate functions in Sobolev spaces H([0, 1]). As a result, given N data, the least-squares estimate ĝ built from P scrambled wavelets has excess risk ||f∗ − ĝ||P = O(||f||Hs([0,1]d)(logN)/P + P (logN)/N) for target functions f∗ ∈ H([0, 1]) of smoothness order s > d/2. An interesting aspect of the resulting bounds is that they do not depend on the distributionP from which the data are generated, which is important in a statistical regression setting considered here. Randomization enables to adapt to any possible distribution. We conclude by describing an efficient numerical implementation using lazy expansions with numerical complexity Õ(2N logN + N), where d is the dimension of the input space.