Quantifying Text Difficulty with Automated Indices of Cohesion and Semantics

We evaluated the effectiveness of new indices of text comprehension in measuring relative text difficulty. Specifically, we examined the efficacy of automated indices produced by the web-based computational tool Coh-Metrix. In an analysis of 60 instructional science texts, we divided texts into groups that were considered to be more or less difficult to comprehend. The defining criteria were based on Coh-Metrix indices that measure independent factors underlying text coherence: referential overlap and vocabulary accessibility. In order to validate the text difficulty groups, participants read and recalled two “difficult” and two “easy” texts that were similar in topic and length. Easier texts facilitated faster reading times and better recall compared to difficult texts. We discuss the implications of these results in the context of theoretically motivated techniques for improving textbook selection.