Text Categorization Using Adaptive Contexttreesjean - Philippe

D epartement de math ematiques et applications-Abstract. A new way of representing texts written in natural language is introduced, as a conditional probability distribution at the letter level learned with a variable length Markov model called adaptive context tree model. Text categorization experiments demonstrates the ability of this representation to catch information about the semantic content of the text. 1. Introduction Managing the information contained in increasingly large textual databases, including corporate databases, digital libraries or the World Wide Web, is now a challenge with huge economic stakes. The starting point of any information organization and management system is a way to transform texts, i.e. long strings of ASCII symbols, into objects adapted to further processing or operations for any particular task. Consider for example the problem of text categorization, that is the automatic assignment of natural language texts to predeened classes or categories. This problem received much attention recently and many algorithms have been proposed and evaluated, including