Using the crowd for readability prediction

Inspired by previous work on crowdsourcing we investigate two different methodologies to assess the readability of a wide variety of text material by implementing two assessment tools. A lightweight crowdsourcing tool which invites users to provide pairwise comparisons and a more advanced version where experts can rank a batch of texts based on readability. In order to validate this approach, readability assessments for a corpus of written Dutch generic texts were gathered. By collecting multiple assessments per text, we explicitly wanted to level out the reader’s background knowledge and attitude. Our findings show that the assessments collected through both annotation tools are highly consistent and that crowdsourcing is a viable alternative to expert labeling. By performing a set of basic machine learning experiments, we further illustrate how the collected data can be used to perform text comparisons or to assign an absolute readability score to an individual text. In order to account for the latter case, we defined a readability measure which is easy to estimate from the data. We do not focus on optimizing the algorithms to achieve the best possible results for the learning tasks, but only carry them out them to illustrate the various possibilities of our data sets. Nevertheless, we show that for each of the tasks, there is a machine learning algorithm outperforming the classical readability formulas. We conclude that readability assessment by comparing texts is a polyvalent methodology, which can be adapted to specific domains and target audiences if required.