OBJECTIVE The purpose of this study was to evaluate Family Medicine Clerkship students' writing skills using an anchored scoring rubric. In this study, we report on the assessment of a current scoring rubric (SR) used to grade written case description papers (CDP) for medical students, describe the development of a revised SR with examination of scoring consistency among faculty raters, and rep...

This study assessed the intra and inter-rater reliability of a modified weight-bearing lunge measure of ankle dorsiflexion range of movement. Thirteen healthy subjects were recruited. Each subject performed 3 repetitions of the lunging method with one rater and 3 more repetitions with a second rater within 30 min. The process was repeated within 3 h. Intra-rater reliability results indicated ex...

We present a system for automated phonetic clustering analysis of cognitive tests of phonemic verbal fluency, on which one must name words starting with a specific letter (e.g., ‘F’) for one minute. Test responses are typically subjected to manual phonetic clustering analysis that is labor-intensive and subject to inter-rater variability. Our system provides an automated alternative. In a pilot...

Abstract In this paper we report on the analyses of alternative approaches to semantic role annotation (FrameNet (FrameNet, 2005), PropBank (Palmer et al., 2005) and VerbNet (Kipper, 2006)) with respect to their models of description; granularity of semantic role sets; definitions of semantic roles concepts; and consistency and reliability of annotations, and we propose a methodological basis f...

BACKGROUND Recent studies of the quality of in-hospital care have used the Quality of Interaction Schedule (QuIS) to rate interactions observed between staff and inpatients in a variety of ward conditions. The QuIS was developed and evaluated in nursing and residential care. We set out to develop methodology for summarising information from inter-rater reliability studies of the QuIS in the acu...

In this paper, we argue that clustering WordNet senses into more coarse-grained groupings results in higher inter-annotator agreement and increased system performance. Clustering of verb senses involves examining syntactic and semantic features of verbs and arguments on a caseby-case basis rather than applying a strict methodology. Determining appropriate criteria for clustering is based primar...

We present methods for evaluating human and automatic taggers that extend current practice in three ways. First, we show how to evaluate taggers that assign multiple tags to each test instance, even if they do not assign probabilities. Second, we show how to accommodate a common property of manually constructed " gold standards " that are typically used for objective evaluation, namely that the...

QUESTIONS What is the intra-rater and inter-rater relative reliability of the Berg Balance Scale? What is the absolute reliability of the Berg Balance Scale? Does the absolute reliability of the Berg Balance Scale vary across the scale? DESIGN Systematic review with meta-analysis of reliability studies. PARTICIPANTS Any clinical population that has undergone assessment with the Berg Balance...

1557-3087/$ – see front matter © 2012 http://dx.doi.org/10.1016/j.teln.2012.06.0 Abstract This article looks at the effectiveness of the educational portfolio in the summative assessment of the nursing student. Findings of high inter-rater reliability point to the usefulness of the portfolio in summative assessment of competency, if accompanied by the appropriate supportive mechanisms. Shortcom...