Weighted kappa is a widely used statistic for summarizing inter-rater agreement on a categorical scale. For rating scales with three categories, there are seven versions of weighted kappa. It is shown analytically how these weighted kappas are related. Several conditional equalities and inequalities between the weighted kappas are derived.The analytical analysis indicates that the weighted kapp...

This paper describes a system for interannotator agreement analysis of ERE annotation, focusing on entity mentions and how the higher-order annotations such as EVENTS are dependent on those entity mentions. The goal of this approach is to provide both (1) quantitative scores for the various levels of annotation, and (2) information about the types of annotation inconsistencies that might exist....

BACKGROUND The Pediatric Balance Scale (PBS) was developed from a modified version of the Berg Balance Scale aiming to obtain a balance scale more appropriate for the child population. OBJECTIVES To adapt the PBS into Brazilian-Portuguese and to evaluate the intra and inter-rater reliability of the Brazilian-Portuguese version of PBS. METHODS To perform the cross-cultural adaptation of the ...

BACKGROUND Global performance rating is frequently used in clinical training despite its known psychometric drawbacks. Inter-rater reliability is low in undergraduate training but better in residency training, possibly because residency offers more opportunities for supervision. The low or moderate predictive validity of global performance ratings in undergraduate and residency training may be ...

One of the primary barriers to the acceptance of surgical simulators is that most simulators still require a significant amount of an instructing surgeon's time to evaluate and provide feedback to the students using them. Thus, an important area of research in this field is the development of metrics that can enable a simulator to be an essentially self-contained teaching tool, capable of ident...

The level of agreement among participants is a key aspect of gesture elicitation studies, and it is typically quantified by means of agreement rates (AR). We show that this measure is problematic, as it does not account for chance agreement. The problem of chance agreement has been extensively discussed in a range of scientific fields in the context of inter-rater reliability studies. We review...

1 This study presented a model-based image-matching (MBIM) motion analysis 2 technique for ankle joint kinematic measurement. Five cadaveric below-hip 3 specimens were manipulated through a full range of ankle joint motions in bare-foot 4 and shoed conditions. The ankle motions were analyzed by bone-pin marker-based 5 motion analysis and MBIM motion analysis techniques respectively. The root me...

This study examined the psychometric properties of the Conduct Problems Risk Screen (CPRS), a seven-item screen derived from DSM-IV-TR criteria that can be completed by parents or teachers. The sample consisted of 4,752 Australian fiveto nine-year-old primary school children. The results showed the parent and teacher screens had very good internal consistency and high convergent validity with o...