BACKGROUND Attention to the problem of adverse events increases worldwide. The present study aimed to evaluate inter-rater reliability of medical record reviews of adverse events in the Thai context. MATERIAL AND METHOD A total of 279 inpatient records were reviewed by 23 clinical auditors. Each record was examined independently by 3 auditors using a standardised review form. Agreements on th...

In this paper, we describe an alternative to the kappa statistic for measuring intercoder agreement. We present a model based on the assumption that the observed surface agreement can be divided into (unknown amounts of) true agreement and chance agreement. This model leads to confidence interval estimates for the proportion of true agreement, which turn out to be comparable to confidence inter...

Currently, computational linguists and cognitive scientists working in the area of discourse and dialogue argue that their subjective judgments are reliable using several different statistics, none of which are easily interpretable or comparable to each other. Meanwhile, researchers in content analysis have already experienced the same difficulties and come up with a solution in the kappa stati...

Model-based inference procedures for the kappa statistic have developed rapidly over the last decade. However, no method has yet been developed for constructing a confidence interval about a difference between independent kappa statistics that is valid in samples of small to moderate size. In this article, we propose and evaluate two such methods based on an idea proposed by Newcombe (1998, Sta...

Currently, computational linguists and cognitive scientists working in the area of discourse and dialogue argue that their subjective judgments are reliable using several diierent statistics, none of which are easily interpretable or comparable to each other. Meanwhile, researchers in content analysis have already experienced the same diiculties and come up with a solution in the kappa statisti...

BACKGROUND Clinical examination of trachoma is used to justify intervention in trachoma-endemic regions. Currently, field graders are certified by determining their concordance with experienced graders using the kappa statistic. Unfortunately, trachoma grading can be highly variable and there are cases where even expert graders disagree (borderline/marginal cases). Prior work has shown that inc...

Although the κ statistic has been used widely as an indicator of rater agreement, there have been some concerns about the existence of different definitions and some peculiar results involving skewed data. This note evaluates different definitions of κ and also demonstrates that the problem with directly comparing κ values, especially for skewed data, can be avoided by comparing their significa...

Kappa (Cohen, 1960) is a popular agreement statistic used to estimate the accuracy of observers. The response of kappa to differing baserates was examined and methods for estimating the accuracy of observers presented. Results suggest that setting a single value of kappa as "minimally acceptable" (i.e., a criterion value) is not useful in ensuring adequate accuracy of observers. Instead, resear...

Cohen’s weighted kappa is a popular descriptive statistic for measuring the agreement between two raters on an ordinal scale. Popular weights for weighted kappa are the linear weights and the quadratic weights. It has been frequently observed in the literature that the value of the quadratically weighted kappa is higher than the value of the linearly weighted kappa. In this paper this phenomeno...