Regression with quadratic loss

Regression with quadratic loss is another basic problem studied in statistical learning theory. We have a random couple Z = (X ,Y ), where, as before, X is anRd -valued feature vector (or input vector) and Y is the real-valued response (or output). We assume that the unknown joint distribution P = PZ = PX Y of (X ,Y ) belongs to some class P of probability distributions over Rd ×R. The learning problem, then, is to produce a predictor of Y given X on the basis of an i.i.d. training sample Z n = (Z1, . . . , Zn) = ((X1,Y1), . . . , (Xn ,Yn)) from P . A predictor is just a (measurable) function f : Rd →R, and we evaluate its performance by the expected quadratic loss L( f ), E[(Y − f (X ))]. As we have seen before, the smallest expected loss is achieved by the regression function f ∗(x), E[Y |X = x], i.e., L∗ , inf f L( f ) = L( f ∗) = E[(X −E[Y |X ])].