Robustness and Stability of Reweighted Kernel Based Regression

Recently it was shown that Kernel Based Regression (KBR) with a least squares loss function may have some undesirable properties from a robustness point of view: even very small amounts of outliers can dramatically affect the estimates. KBR with other loss functions is much more robust, but often gives rise to complicated computations (e.g. for Huber or logistic losses). In classical statistics robustness is often improved by reweighting the original estimate. In this paper we provide a theoretical framework for reweighted KBR and analyze its robustness. Some important differences are found with respect to linear regression, indicating that LS-KBR with a bounded kernel is much more suited for reweighting. Our results give practical guidelines for a good choice of weights, providing robustness as well as fast convergence. In particular a logistic weight function seems an appropriate choice. Robustness is analyzed by means of the influence function. Some interesting links with stability and the leave-one-out-error are explored as well, indicating that reweighting also improves the stability of KBR, especially at heavy-tailed distributions.