Testing Multiple Forecasters∗

We consider a cross-calibration test of predictions by multiple potential experts in a stochastic environment. This test checks whether each expert is calibrated conditional on the predictions made by other experts. We show that this test is good in the sense that a true expert—one informed of the true distribution of the process—is guaranteed to pass the test no matter what the other potential experts do, and false experts will fail the test on all but a small (category one) set of true distributions. Furthermore, even when there is no true expert present, a test similar to cross-calibration cannot be simultaneously manipulated by multiple false experts, but at the cost of failing some true experts. ∗We wish to thank Nabil Al-Najjar, Brendan Beare, Dean Foster, Sergiu Hart, Stephen Morris, Wojciech Olszewski, Alvaro Sandroni, Jakub Steiner, and Jonathan Weinstein for helpful comments and suggestions. The first author gratefully acknowledges the support of the NSF grant IIS-0205633 and the hospitality of the Institute for Advanced Studies at the Hebrew University. †email: [email protected] ‡email: [email protected]