Measures of dispersion for corpus data: an overview, a suggestion, and a research program I

The most frequently used statistic in corpus linguistics are the frequency of occurrence of some linguistic variable and the frequency of co-occurrence of two or more linguistic variables. However, as has been pointed out correctly and repeatedly, frequencies of (co-)occurrence in isolation may sometimes be severely misleading given that they alone to not take into consideration the degree of dispersion of the relevant linguistic variable in question. In order to handle such problems, several scholars suggested a variety of dispersion measures and adjusted frequency measures. Unfortunately, however, few scholars appear to be aware of this issue and measures of dispersion are apparently not widely know and even less widely used. Another unfortunate aspect of such measures is that many of them also come with a variety of problems. I pursue three objectives with this article. First, in order to raise awareness of this thematic complex and make the available measures more widely known, I present an overview of a large number of dispersion measures and adjusted frequency measures – including some more recent measures that have not found their way into research articles and textbooks yet – and summarily discuss some of their advantages and disadvantages. Second, I propose for discussion a conceptually very simple alternative measure, DP, explain and exemplify its properties, and compare it to the more or less established measures on the basis of fictitious distributions from the literature, word frequencies from the BNC Sampler, and co-occurrence data from the ICEGB. I will conclude that DP is at least as discriminatory as most if not all other existing measures but conceptually simpler and even better in some respects.