A Methodologically Improved Study on Raters’ Personality and Rating Severity in Writing Assessment

Personality is an inherent rater’s characteristic influencing rating severity, but very few studies examined their relationship and the findings were inconclusive. This study aimed to re-investigate between raters’ personality severity with more control on relevant variables reliable analysis of severity. Female novice raters ( n = 28) from a demographically homogeneous background recruited rate two occasions essays written by 111 students in intermediate-level Chinese as foreign language program. Raters’ traits measured using complete version NEO-PI-R. Many-faceted Rasch measurement model repeated applied yield robust estimates In addition, order effect was carefully controlled. Extroversion found be positively correlated r(26) .495, p .010. Furthermore, valid predictor t(24) 2.792, .010, R 2 .21, Cohen’s d .77, Hattie’s r .37. Practical implications for developing individualized online rater calibration large-scale writing assessments discussed, followed limitations present study.