Stability of cross-validation and minmax-optimal number of folds

In this paper, we analyze the properties of cross-validation from the perspective of the stability, that is, the difference between the training error and the error of the selected model applied to any other finite sample. In both the i.i.d. and non-i.i.d. cases, we derive the upper bounds of the one-round and average test error, referred to as the one-round/convoluted Rademacher-bounds, to quantify the stability of model evaluation for cross-validation. We show that the convoluted Rademacher-bounds quantify the stability of the out-of-sample performance of the model in terms of its training error. We also show that the stability of cross-validation is closely related to the sample sizes of the training and test sets, the ‘heaviness’ in the tails of the loss distribution, and the Rademacher complexity of the model class. Using the convoluted Rademacher-bounds, we also define the minmax-optimal number of folds, at which the performance of the selected model on new-coming samples is most stable, for cross-validation. The minmax-optimal number of folds also reveals that, given sample size, stability maximization (or upper bound minimization) may help to quantify optimality in hyper-parameter tuning or other learning tasks with large variation.