Keyword Extraction for Text Characterization

Keywords are valuable means for characterizing texts. In order to extract keywords we propose an efficient and robust, language-and domain-independent approach which is based on small word parts (quadgrams). The basic algorithm can be improved by reexamining and re-ranking keywords using edit distance (i.e. Levenshtein distance) and an algorithm based on the relativistic addition of velocities (here: weights). For the purpose of evaluation, we compare our approach to frequency-based keyword extraction (exemplary text collection: 45000 intranet documents in German and English). The analysis of huge text collections usually aims at finding relevant texts (known as text retrieval with search engines) or text groups (supervised grouping like categorization or classifying, unsupervised grouping like clustering). Thus, all these text mining tasks result in retrieved texts or text groups. For an overview of text mining tasks see [Ch00]. But it is a tedious task of any information-seeking user to scan all retrieved items. In order to facilitate this task, most text mining systems characterize their resulting texts with various kinds of annotations. They link texts to external topic schemes or find relevant concepts out of the texts themselves. These items of external topic schemes as well as text-based concepts can be presented as keywords which are a helpful characterization of textual content. An extensive survey of summarization gives [Ho02]. Here, topic identification as the simplest type of a summary also subsumes keyword extraction.