Kappa Statistics for Multiple Raters Using Categorical Classifications

In order to assess the reliability of a given characterization of a subject it is often necessary to obtain multiple readings, usually but not always from different individuals or raters. The degree of agreement among the various raters gives some indication as to the consistency of the values. If agreement is high, we feel more confident the ratings reflect the actual circumstance. If agreement among the raters is low, we are less confident in the results. While several methods are available for measuring agreement when there are only two raters, this paper concentrates on presenting a generalized implementation of the Fleiss (1981) technique. This method can be utilized even in situations where there are more than two raters and/or categories. A review of the statistical theory behind the intraclass correlation coefficients and kappa statistics obtained when looking at the above situations is presented. SAS code is provided which utilizes basic SAS procedures. INTRODUCTION Often we are faced with determining the measurement of interrater agreement when the ratings are on a categorical scale. When the number of raters is equal to two, this is easily accomplished by using SAS PROC CORR to get an estimate of the correlation coefficient. SAS 6.10 PROC FREQ with the AGREE option also provides an easy way to obtain the kappa statistic when there are only two raters. Fleiss describes a technique for obtaining interrater agreement when the number of raters is greater than or equal to two. This paper concentrates on the ability to obtain a measure of agreement when the number of raters is greater than two. It also concentrates on the technique necessary when the number of categories into which the ratings can fall is greater than two. BACKGROUND The data used in this paper to demonstrate the technique of calculating interrater agreement was compiled from a gastric graft versus host disease study. (Washington, et al, 1996, submitted). This research compares the results of three pathologists’ diagnoses of 51 different gastric biopsies. Each pathologist reviewed the gastric biopsies in a blinded fashion. The specific data used in the example in this paper involves degree of agreement on density of the inflammatory infiltrate in the lamina propria. The categories of degree of inflammation range from zero to three. METHODS Since the kappa statistic ( $ k ) was first proposed by Cohen (1960), variants have been proposed by others, including Scott (1955), Maxwell and Pilliner (1968), and Bangdiwala (1987). SAS code has also been presented by Gaccione (1993) to compute the kappa statistic. The various