Estimating the Intra-Rater Reliability of Essay Raters

The intra-rater reliability in rating essays is usually indexed by the inter-rater correlation. We suggest an alternative method for estimating intra-rater reliability, in the framework of classical test theory, by using the dis-attenuation formula for inter-test correlations. The validity of the method is demonstrated by extensive simulations, and by applying it to an empirical dataset. It is recommended to use this estimation method whenever the emphasis is not on the average intra-reliability of a group of raters, but when the intrarater reliability of a specific rater is of interest, e.g., when the error-variance component of the scores is of interest in order to estimate true scores.