Emerging studies have shown that observers' ratings of personality predict performance behaviors better than do self-ratings. However, it is unclear whether these predictive advantages stem from (a) use of observers who have a frame of reference more closely aligned with the criterion ("narrower scope") or (b) observers having greater accuracy than targets themselves ("clearer lens"). In a prim...

Frame-of-reference training has been shown to be an effective intervention for improving the accuracy of performance ratings (e.g., Woehr & Huffcutt, 1994). Despite evidence in support of the effectiveness of frame-of-reference training, few studies have empirically addressed the ultimate goal of such training, which is to teach raters to share a common conceptualization of performance (Athey &...

The study investigates impressions formed through social networking sites, specifically the initial judgments we make of others when first momentarily exposed to their photograph. The personality characteristics of 52 Female Facebook profile owners were evaluated by a group of raters who briefly viewed the current profile picture of each person. Analysis revealed consensus between raters when j...

BACKGROUND Telephone triage is a core but vulnerable part of the care process at out-of-hours general practitioner (GP) cooperatives. In the Netherlands, different instruments have been used for assessing the quality of telephone triage. These instruments focussed mainly on communicational aspects, and less on the medical quality of triage decisions. Our aim was to develop and test a minimum se...

Reflection is widely accepted as a learning tool and is considered integral to professional practice. Journal writing is advocated in facilitating reflection, yet little is written about how to assess reflection in journals. The purpose of this study was to develop and test a method of assessing the elements of reflection in journals and to determine whether, and to what level, reflection occur...

Making judgments about people is a difficult task, and various efforts have been made to achieve greater accuracy in the judgments. The most popular attempt has been the adoption of Likert-type (typically five point) scales for rating people on certain dimensions or competencies. In order to achieve any degree of accuracy from these Likert scaling procedures, considerable training of all raters...

The volume of hypointense lesions on T1 weighted brain MRI represents an increasingly used MR endpoint in phase III MS treatment trials. In this study we evaluated the reproducibility of hypointense T1 lesion volume quantification in a cohort of Multiple Sclerosis (MS) patients. The gadolinium enhanced T1 weighted brain MR images of 33 MS patients from three European centers were used in this s...

Sample size requirements that achieve a prespecified expected lower limit for a confidence interval about the intraclass kappa statistic are supplied for the case of multiple raters and a binary outcome variable. The expected lower confidence limit achievable for a given number of subjects and raters is also presented. These results should be useful in the planning stages of an interobserver ag...

It is common practice to rely on a convenience sample of subject matter experts (SMEs) when developing scoring keys for situational judgment tests (SJTs). However, the defining characteristics what constitutes SME are often ambiguous and inconsistent. Sampling SMEs can also impose considerable costs. Other research fields have adopted crowdsourcing methods replace or reproduce judgments thought...

In this study we determined whether untrained raters’ assessments of fluency in low-proficiency second language speech were related to temporal measures and whether they varied across tasks. We collected speech samples from 20 beginner Mandarin learners of English on picture description, monologue, and dialogue tasks. Temporal measures were made on each sample. Twenty-eight untrained judges rat...