INTEX: A Corpus Processing System

INTEX is a text processor; it is usually used to parse corpora of several megabytes. It includes several built-in large coverage dictionaries and grammars represented by graphs; the user inay add his/her own dictionaries and gramnlars. These tools am applied to texts in order to locate lexical and syntactic patterns, remove ambiguities , and tag words. INTEX builds collcordances and indexes of all types of patterns; it is used by linguists to analyse corpora, but can also be viewed as an information feb'lewd system. INTROI)UCTION INTEX automatically identities words and mor-pho-syntactic patterns in large texts. By using INTEX, one can: .... build the dictionary of lhe words of the texts; words may be simple words (sequences of letters, e.g. table), compounds (sequences of simple words which include a separator, e.g. worU pro~ cessor) or complete expressions (sequences of words which accept insertions, bucket);-locate in texts all occurrences of a given word (even if inflected), a given category (e.g. all feminine plural adjectives) or a morpho-syniactic pat~ tern (a regular expression);-apply grammars represented by recursive graphs to texts; build indexes or concordances for all occurrences of the previous patterns; .... use local grammars to remove word and uller-ance ambiguities in texts, or to detect errors or deviant sequences. While INTEX already i,lcludes several built-in dictionaries and granunars, it allows tile user to create, c(lit and add his/her own tools, hi order to increase coverage of texts and to remove additional ambiguilies. The user th'st loads a text and selects the woi'kiug langl.iage I. INT[~.X counls lhc nulnbor of lokens in the lexl, lhe number of different ones, and sorts lhoni by frequency. Theil Ihe user selects linguistic tools to parse the text. Tools aye either diclio.. nnries or tinilO stale transducers (FSTs). INTEX is based on lwo large coverage builtqn dictionaries:-the I)IT:.LAI ~ diclio,mry contains over 700,000 simple words, basically all the simple words of the language 2. Each entry in the I)ELAI: is asso. cialed wilh explicit morphological infornlathm for each word: its canonical form (e.g. the intini-live for verbs), its part of speech (e.g. Noun), aud some inllectional information (e.g. th'st person singular present). I lere are three entries of the