Running Head: RATING DIFFERENCES IN MULTI-RATER FEEDBACK Rating Differences in Multi-Rater Feedback: A New Look at an Old Issue

"Three hundred and sixty-degree feedback" ("360") is the popular name for performance feedback collected from multiple raters. In the typical 360 process, supervisor(s), subordinates, peers, and (less frequently) internal or external customers provide feedback on performance for each target ratee, using some type of standardized instrument (London & Smither, in press; Tornow, 1993a). The ratee then is expected to use the data, along with his/her self-ratings, to make appropriate behavioral changes to improve performance (London & Smither, in press). Multi-rater feedback has assumed a substantial role in U.S. organizations in the last decade (O'Reilly, 1994). In spite of its rapid growth, however, much remains to be learned about its application and interpretation (London & Smither, in press). While there have been a number of studies documenting the lack of agreement among rater groups (London, 1995), few have focused at the level of dimensions rather than overall ratings. Additionally, few have attempted to determine why a lack of agreement exists (Cardy & Dobbins, 1994). The purpose of this research, therefore, is to contribute to a broader understanding of 360 and rating differences among different rater groups. This paper will begin by exploring the 360 process. The research on the level of agreement that has been exhibited among rater groups will be presented. Finally, a study to explore rater agreement at the dimension level will be reviewed. 360 is a complex, multi-step process. Based upon a literature review, a model (Figure 1) was developed to illustrate key components of the 360 process, and to provide structure to the broad spectrum of literatures with relevance to 360. This process model will now be explored. Purpose. The first component in Figure 1's process model is the purpose for which multi-rater feedback is to be used. 360 has been used by researchers and practitioners to address a variety of individual and organizational goals. They include: 1. To improve the subjective measurement of performance. Although supervisory ratings are widely used for measuring performance, they are subject to a variety of intentional and unintentional errors (Cardy & Dobbins, 1994). 360 has Abstract Understanding rater disagreement in multi-rater (360) feedback efforts is important to both scientists and practitioners. This study used structural equations modeling to test for the presence of (a) construct definition differences and (b) rating scale point differences in each of sixteen performance dimensions. Data used were from the Center for Creative Leadership's "Benchmarks ® " …