Annotation Curricula to Implicitly Train Non-Expert Annotators

Abstract Annotation studies often require annotators to familiarize themselves with the task, its annotation scheme, and data domain. This can be overwhelming in beginning, mentally taxing, induce errors into resulting annotations; especially citizen science or crowdsourcing scenarios where domain expertise is not required. To alleviate these issues, this work proposes curricula, a novel approach implicitly train annotators. The goal gradually introduce task by ordering instances annotated according learning curriculum. do so, formalizes curricula for sentence- paragraph-level tasks, defines an strategy, identifies well-performing heuristics interactively trained models on three existing English datasets. Finally, we provide proof of concept carefully designed user study 40 voluntary participants who are asked identify most fitting misconception tweets about Covid-19 pandemic. results indicate that using simple heuristic order already significantly reduce total time while preserving high quality. thus promising research direction improve collection. facilitate future research—for instance, adapt specific tasks expert scenarios—all code from consisting 2,400 annotations made available.1