Abstract Hate speech annotation for training machine learning models is an inherently ambiguous and subjective task. In this paper, we adopt a perspectivist approach to data annotation, model evaluation hate classification. We first focus on the process argue that it drastically influences final quality. then present three large datasets incorporate annotator disagreement use them train evaluat...