Identifying Sources of Inter-Annotator Variation: Evaluating Two Models of Argument Analysis
نویسنده
چکیده
This paper reports on a pilot study where two Models of argument were applied to the Discussion sections of a corpus of biomedical research articles. The goal was to identify sources of systematic inter-annotator variation as diagnostics for improving the Models. In addition to showing a need to revise both Models, the results identified problems resulting from limitations in annotator expertise. In future work two types of annotators are required: those with biomedical domain expertise and those with an understanding of rhetorical structure.
منابع مشابه
Annotating Discourse Connectives And Their Arguments
This paper describes a new, large scale discourse-level annotation project – the Penn Discourse TreeBank (PDTB). We present an approach to annotating a level of discourse structure that is based on identifying discourse connectives and their arguments. The PDTB is being built directly on top of the Penn TreeBank and Propbank, thus supporting the extraction of useful syntactic and semantic featu...
متن کاملEvaluating of the efficiency of AMMI and BLUP models and their integration for identifying high-yielding durum wheat (Triticum turgidum L. var. durum) genotypes adapted to warm rainfed regions of Iran
The aim of this study was to evaluate the efficiency of yield stability analysis models and to assess genotype × environment interaction effect on grain yield of 20 durum wheat genotypes for identifying high yielding and adapted genotypes by BLU and AMMI models using experimental data of four cropping cycles (2009-2013) in five filed stations in warm rainfed regions of Iran. The results of Like...
متن کاملEvaluating Inter-Annotator Agreement on Historical Spelling Normalization
This paper deals with means of evaluating inter-annotator agreement for a normalization task. This task differs from common annotation tasks in two important aspects: (i) the class of labels (the normalized wordforms) is open, and (ii) annotations can match to different degrees. We propose a new method to measure inter-annotator agreement for the normalization task. It integrates common chancec...
متن کاملInter-sentential Coreferences in Semantic Networks: An Evaluation of Manual Annotation
We present an evaluation of inter-sentential coreference annotation in the context of manually created semantic networks. The semantic networks are constructed independently be each annotator and require an entity mapping priori to evaluating the coreference. We present the model used for mapping the semantic entities as well as the algorithm used for our evaluation task. We present the raw sta...
متن کاملCharacter Identification on Multiparty Conversation: Identifying Mentions of Characters in TV Shows
This paper introduces a subtask of entity linking, called character identification, that maps mentions in multiparty conversation to their referent characters. Transcripts of TV shows are collected as the sources of our corpus and automatically annotated with mentions by linguistically-motivated rules. These mentions are manually linked to their referents through crowdsourcing. Our corpus compr...
متن کامل