Holistic Annotation of Discourse Coherence Quality in Noisy Essay Writing

In this paper, we describe a holistic annotation scheme for coherence quality that requires little expertise on the part of the human annotator; we also present computational systems for evaluating coherence quality in essays. A formidable challenge to reliability when annotating discourse coherence comes from differences among annotators in the inferences that they draw when reading an essay. This may reflect differences in their background knowledge or in their willingness to bridge what might otherwise seem to be disconnected portions of the text. Despite these differences, we achieved adequate reliability for a holistic binary coherence annotation of essays written for five different types of large-scale assessment. When designing computational systems to score these essays for coherence quality, we faced a number of issues not encountered in most previous work, which has focused primarily on well-formed text. Foremost among these was the need to develop models based on features that reflect the coherence quality criteria which are found in human essay scoring guides, including aspects of writing quality (e.g., the presence of grammatical errors) that might interfere with constructing the meaning of the essay. Such features are needed to produce meaningful scores and to provide the basis for instructional feedback to the student or test-taker. We present results of testing various computational models on essays using the binary discourse coherence score.