Nonlocal Language Modeling based on Context Co-occurrence Vectors

This paper presents a novel nonlocal language model which utilizes contextual information. A reduced vector space model calculated from co-occurrences of word pairs provides word co-occurrence vectors. The sum of word cooccurrence vectors represents the context of a document, and the cosine similarity between the context vector and the word co-occurrence vectors represents the long-distance lexical dependencies. Experiments on the Mainichi Newspaper corpus show signi cant improvement in perplexity (5.0% overall and 27.2% on target vocabulary)