Beyond Majority Voting: Generating Evaluation Scales using Item Response Theory

We introduce Item Response Theory (IRT) from psychometrics as an alternative to majority voting to create an IRT gold standard (GSIRT ). IRT describes characteristics of individual items in GSIRT their difficulty and discriminating power and is able to account for these characteristics in its estimation of human intelligence or ability for an NLP task. In this paper, we evaluated IRT’s model-fitting of a majority vote gold standard designed for Recognizing Textual Entailment (RTE), denoted as GSRTE . By collecting human responses and fitting our IRT model, we found that up to 31% of GSRTE were not useful in building GSIRT for RTE. In addition, we found low inter-annotator agreement for some items in GSRTE suggesting that more work is needed for creating intelligent gold-standards.