Calibrating Prediction Regions

Suppose the variable X to be predicted and the learning sample Y" that was observed have a joint distribution, which depends on an unknown parameter 0. The parameter 0 can be finite or infinite dimensional. A prediction region Dn for X is a random set, depending on Yn, that contains X with prescribed probability a. This paper studies methods for controlling simultaneously the conditional coverage probability of Dn, given Yn, and the overall (unconditional) coverage probability of Dn. The basic construction yields a prediction region Dn which has the following properties in regular models: Both the conditional and overall coverage probabilities of Dn converge to a as the size n of the leaming sample increases. The convergence of the former is in probability. Moreover, the asymptotic distribution of the conditional coverage probability about a is typically normal; and the overall coverage probability tends to a at rate ni1. Can one reduce the dispersion of the conditional coverage probability about a and increase the rate at which overall coverage probability converges to a? Both issues are addressed. The paper establishes a lower bound for the asymptotic dispersion of conditional coverage probability. The paper also shows how to calibrate Dn so as to make its overall coverage probability converge to a at the faster rate n-2. This calibration adjustment does not affect the asymptotic distribution or dispersion of the conditional coverage probability, in a first-order analysis. In general, a bootstrap Monte Carlo algorithm accomplishes the calibration of Dn. In special cases, analytical calibration is possible. * This research was supported in part by NSF Grant DMS 87-01426. Part of the work was done while the author was a guest of Sonderforschungsbereich 123 at Universitgt Heidelberg. The author thanks G. Sawitzki and F. Seillier for helpful comments.