Bagging Does Not Always Decrease Mean Squared Error

Bagging is a device intended for reducing the prediction error of learning algorithms. In its simplest form, bagging draws bootstrap samples from the training sample, applies the learning algorithm to each bootstrap sample, and then averages the resulting prediction rules. Heuristically, the averaging process should reduce the variance component of the prediction error. This is supported by empirical evidence suggesting that bagging can indeed reduce prediction error and appears to be most effective for cart trees, which are highly unstable functions of the data. We study the effects of bagging for the simple class of U-statistics. While these do not describe cart trees, U-statistics have the advantage of admitting a complete and rigorous analysis. We find that bagging always inceases bias, but the effects on variance and mean squared error depend on the specifics of the U-statistic and its distribution. We also find a correspondence to order 1/N2 for bagging based on resampling with and without replacement, respectively. AT&T Labs–Research, 180 Park Ave, Florham Park, NJ 07932-0971; [email protected] Department of Statistics, University of Washington, Seattle, WA 98195-4322; [email protected]. Research partially supported by NSF grant DMS 9803226. This work was performed while the second author was on sabbatical leave at AT&T Labs.