Analysis of the oracle risk in multi-task ridge regression

In this talk1 we will present new results obtained in the multiple kernel ridge regression framework, which we refer to as multi-task regression. Multi-task techniques come into play when experimental limitations makes it impossible to increase the sample size n, which is the classical way to improve the performance of the estimators. However, it is often possible to have access to other closely related regression problems. Then, taking into account the similarity between those p different tasks and treating them in a multi-dimensional setting may lead to some improvement of the procedure, hoping that the heuristic “p tasks with each n observations is equivalent to an n× p sample” is valid. In this setting we use a generalization of the ridge regression procedure proposed by Evgeniou et al. [2005], in which a penalization term containing a matricial parameter expresses the similarity between the tasks. A fully data-driven selection of this matrix was proposed by Solnon et al. [2012], resulting in an oracle inequality, which shows that the selected estimator attains a risk that is close to the best possible risk, the oracle risk. We will show how we obtain an estimation of this multi-task oracle risk. Then, we find several settings in which the single-task oracle risk can also be expressed and compare both risks. This allows us to discriminate several settings, depending on whether the multi-task oracle outperforms the single-task oracle, the opposite, or whether both behave similarly. We also show that our estimation of the multi-task oracle risk is precise enough to be plugged in the latter oracle inequality, thus showing that the estimator of Solnon et al. [2012] has a lower risk than the single-task oracle risk in favorable settings. We finally show simulation results in situations where the oracle risk can no longer be explicitly computed and show that, in those cases, the oracle retains the same virtues and disadvantages as before.