Educators often employ various training techniques to reduce raters’ subjectivity. Negotiation is a technique which can assist novice raters to co-construct a shared understanding of the writing assessment when rating collaboratively. There is little research, however, on rating behaviors of novice raters while employing negotiation techniques and the effect of negotiation on their understandin...

Creativity research crucially relies on creativity evaluations by external raters, but it is not clear what properties characterize good raters. In the present study, we investigated whether rater personality and are related to discernment (i.e., ability distinguish creative from uncreative responses) when evaluating others’ responses. To this end, 166 novice raters evaluated a pre-selected set...

OBJECTIVES To develop comprehensive and reliable radiation-free methods to quantify femoral and acetabular morphology using MRI. METHODS 32 hips (16 subjects, 6 with intra-articular hip disorder (IAHD); 10 controls) were included. A 1.5T magnetic resonance system was used to obtain 3D fat suppressed gradient echo images at the pelvis and distal femora. After acquisition, pelvic images were po...

OSCEs (Objective Structured Clinical Examinations) are widely used in health professions to assess clinical skills competence. Raters use standardized binary checklists (CL) or multi-dimensional global rating scales (GRS) to score candidates performing specific tasks. This study assessed the reliability of CL and GRS scores in the assessment of veterinary students, and is the first study to dem...

CONTEXT Landing kinematics have been identified as a risk factor for knee injury. Detecting atypical kinematics in clinical settings is important for identifying individuals at risk for these injuries. OBJECTIVE To determine the reliability of a handheld tablet and application (app) for measuring lower-extremity kinematics during drop vertical-jump landings. DESIGN Measurement reliability. ...

Although the use of verbal protocols is growing in oral assessment, research on the use of raters’ verbal protocols is rather rare. Moreover, those few studies did not use a mixed-methods design. Therefore, this study investigated the possible impacts of rater training on novice and experienced raters’ application of a specified set of standards in rating. To meet this o...

Recent literature suggests a revival of interest in single-case methodology (e.g., the randomised n-of-1 trial is now considered Level 1 evidence for treatment decision purposes by the Oxford Centre for Evidence-Based Medicine). Consequently, the availability of tools to critically appraise single-case reports is of great importance. We report on a major revision of our method quality instrumen...

OBJECTIVE The purpose of this study was to determine the inter-rater reliability of ultrasound imaging for assessing trunk muscle morphologic characteristics at rest and while contracted among different pairs of novice raters. The secondary purpose was to compare 3 different measurement techniques for assessing lateral abdominal muscle thickness. METHODS A single-group repeated measures relia...

Study Design Reliability study. Purpose To examine the reliability of novice and experienced raters for measurements of the size and composition of the cervical extensor muscles using a thresholding technique. Overview of Literature Although some authors have reported on the dependability of magnetic resonance imaging (MRI) measurements of the cervical muscles, there remains some variabilit...

A new method to characterize the degree of fire damage to gypsum wallboard is introduced, implemented, and tested to determine the efficacy of its application among novices. The method was evaluated by comparing degree of fire damage assessments of novices with and without the method. Thirty-nine "novice" raters assessed damage to a gypsum wallboard surface, completing 66 ratings, first without...