Correlation and Large-Scale Simultaneous Significance Testing

Large-scale hypothesis testing problems, with hundreds or thousands of test statistics “zi” to consider at once, have become familiar in current practice. Applications of popular analysis methods such as false discovery rate techniques do not require independence of the zi’s, but their accuracy can be compromised in high-correlation situations. This paper presents computational and theoretical methods for assessing the size and effect of correlation in large-scale testing. A simple theory leads to the identification of a single omnibus measure of correlation for the zi’s order statistic. The theory relates to the correct choice of a null distribution for simultaneous significance testing, and its effect on inference.