Active Comparison of Prediction Models (Online Appendix)

Let ∆̂n,q = ∑n i=1 wi`i andWn = ∑n i=1 wi with wi = p(xi) q(xi) and `i = `(f1(xi), yi)−`(f2(xi), yi). We note that for examples drawn according to q(x), E[∆̂n,q] = n∆ and E[Wn] = n. The random variables w1, . . . , wn and w1`1, . . . , wn`n are iid, therefore the central limit theorem implies that 1 n∆̂ 0 n,q and 1 nWn are asymptotically normally distributed with √ n ( 1 n ∆̂n,q −∆ ) n→∞ −→ N (0,Var[wi`i]) √ n ( 1 n Wn − 1 ) n→∞ −→ N (0,Var[wi]),