Instance-by-instance optimal identity testing

We consider the problem of verifying the identity of a distribution: Given the description of a distribution over a discrete support p = (p1, p2, . . . , pn), how many samples (independent draws) must one obtain from an unknown distribution, q, to distinguish, with high probability, the case that p = q from the case that the total variation distance (L1 distance) ||p− q||1 ≥ ε? We resolve this question, up to constant factors, on an instance by instance basis: there exist universal constants c, c′ and a function f(p, ε) on distributions and error parameters, such that our tester distinguishes p = q from ||p− q||1 ≥ ε using f(p, ε) samples with success probability > 2/3, but no tester can distinguish p = q from ||p− q||1 ≥ c · ε when given c′ · f(p, ε) samples. The function f(p, ε) is upper-bounded by a multiple of ||p||2/3 ε2 , but is more complicated, and is significantly smaller in cases when p has many small domain elements, or a single large one. This result significantly generalizes and tightens previous results: since distributions of support at most n have L2/3 norm bounded by √ n, this result immediately shows that for such distributions, O( √ n/ε) samples suffice, tightening the previous bound of O( √ npolylog n ε4 ) for this class of distributions, and matching the (tight) known results for the case that p is the uniform distribution over support n. The analysis of our very simple testing algorithm involves several hairy inequalities. To facilitate this analysis, we give a complete characterization of a general class of inequalities— generalizing Cauchy-Schwarz, Hölder’s inequality, and the monotonicity of Lp norms. Specifically, we characterize the set of sequences a = a1, . . . , am, b = b1, . . . , bm, c = c1 . . . , cm, for which it holds that for all finite sequences of positive numbers x = x1, . . . and y = y1, . . . , ∏