Statistical Consistency of Keywords Dictionary Parameters

The construction of optimal keywords dictionary is one of the most important tasks in developing of thesaurusbased Informational Retrieval Systems and some other NLP applications. The problem is, which parameters of keywords dictionaries may be rated as consistent ones from statistical point of view (that is not dependent on sample size)? Analysis of the recent scientific works and the results of our own investigations allowed us to determine a rather complete list of parameters, which may be used for description of texts and language resources. Each of these parameters was exposed to the consistency test. Methodology for consistency test has been elaborated using the method of least squares with a number of principle modifications. Our main results are the following: 1) Theoretically all analyzed parameters have either upper or lower limits. That means that in principle they are statistically consistent. However, for the most of parameters actual consistency is achieved only in the very big sample sizes. 2) The most consistent parameters turned out to be: order coefficient, logarithmic concentration coefficient, entropy, rank golden section, and rank mean. Their rapid speed of convergence to the limit values allows to effectively perform classification procedures on data of the arbitrary size. 3) The proposed model of approximation makes it possible to forecast the values of all parameters for any sample size.