Errata for SemEval-2013 Task 13: Word Sense Induction for Graded and Non-Graded Senses

SemEval-2013 Task 13 measures the performance of Word Sense Induction (WSI) and unsupervised Word Sense Disambiguation (WSD) systems. Given a word in context, the systems must label the word with one or more senses, indicating valid interpretations of the word, and where each sense annotation may be accompanied by a weight indicating how likely that interpretation is. Before performing the annotation task, WSI systems first induce the different meanings of a word by examining usages of the word in text; in contrast, the WSD systems were asked to use the WordNet 3.1 sense inventory. After the completion of Task 13 and the publication of the task description paper (Jurgens and Klapaftis, 2013), a software bug was discovered in the evaluation program that affected the scores in a limited set of circumstances.1 Specifically, the bug resulted in an incorrect calculation of Recall for a WSI or WSD system when not all instances were labeled with senses. In most cases, a system does label all instances with senses and thus, the bug does not occur. However, the task report includes a follow-up experiment that tested systems using only those instances that were labeled with multiple senses; in this setting, many WSI systems ultimately reported fewer instances and, due to the bug, had incorrect scores in the task report.2