Corpus-Based Stemming using Co-occurrence of Word Variants

Stemming is used in many information retrieval (IR) systems to reduce variant word forms to common roots. It is one of the simplest applications of natural language processing to IR, and one of the most eeective in terms of user acceptance and consistent, though small, retrieval improvements. Current stemming techniques do not, however, reeect the language use in spe-ciic corpora and this can lead to occasional serious retrieval failures. We propose a technique for using corpus-based word variant co-occurrence statistics to modify or create a stemmer. The experimental results generated using English newspaper and legal text and Spanish text demonstrate the viability of this technique and its advantages relative to conventional approaches.