Comparative Analysis of Bradley-Terry and Thurstone-Mosteller Paired Comparison Models for Image Quality Assessment

In image quality assessment, preference for various image processing algorithms or treatments is often determined using paired comparisons. In this experimental design, pairs of images processed by different algorithms or “treatments” are presented to a judge. The preferred treatment is selected and a tally is kept of the number of times each treatment is preferred to another. It is possible to estimate an interval scale for treatments in a hypothetical psychological space using this method. There are two dominate paired comparison statistical models: Thurstone-Mosteller Case V (TM) (corresponding to Thurstone’s Law of Comparative Judgment, Case V) and Bradley-Terry (BT). Although TM is used almost exclusively in the imaging literature, the BT formulation is more mathematically developed. Owing to its parsimony, it provides tractable maximum-likelihood estimators for scales, simultaneous confidence intervals and hypothesis tests for model fit, uniformity, and differences among populations of judges. In practice, TM and BT yield nearly identical scale estimates for complete data. In some experimental designs, many treatments are compared. Owing to the large number of possible treatment pairs, not every comparison is made, leading to an incomplete matrix of preference counts. Unlike TM, BT model applies directly to incomplete data under mild restrictions We compare and critique TM and BT models. Statistical analyses, many not available under TM, are demonstrated. An argument is made that BT offers overwhelming advantages to the imaging community and should be used instead of TM. Introduction This paper compares two well-known paired comparison models: the Thurstone-Mosteller (TM) model (by which we mean Thurstone’s Law of Comparative Judgment, Case V) and the Bradley-Terry (BT) model. (Mosteller’s name is included in TM due to his work on the statistical analysis of Thurstone’s model). We argue here that BT model should be used in place of TM because presently the former is more developed mathematically than the latter. In particular, easy formulas exist for maximum likelihood estimates (mle) of scale parameters. The asymptotic theory of mle’s yields estimators for confidence regions and test statistics based on likelihood ratios for hypothesis testing. TM is privileged within the imaging community ostensibly owing to its origins in psychophysics. Yet it is universally acknowledged that TM and BT yield similar scale estimates. The theory (and software) for generalized linear models can produce mle’s yet BT, with its roots in experimental design and consumer choice modeling, offers numerically easier statistical procedures. We present no new research although we do show an alternative analysis to previously published data. Our intent is to provide the imaging community with a general context for paired comparisons, compare and contrast the two models, and demonstrate the advantages of BT. The Linear Model TM and BT models are both linear models of paired comparisons. In such models, probabilities of preference can be mapped to scales. Formally (following David, 1988 [4]), let i V and j V represent “merits” of objects i A and j A , respectively. In a psychophysics setting, the i V might represent sensation magnitudes on a scale. We represent the observed merit of object i A by random variable i X owing to observation-to-observation variation. A linear model takes the form ( ) ( ) i j i j ij P X X H V V π > ≡ = − (1) where H is a monotonic, increasing function such that ( ) 0 H −∞ = , ( ) 1 H +∞ = , and ( ) 1 ( ) H x H x − = − . There are obviously an infinite number of choices for function H, the two of concern here are the Thurstone-Mosteller model where H is the normal cumulative distribution function with zero mean and the Bradley-Terry model where [ ] 1 ( ) 1 tanh( / 2) 2 H x x = + (2) The task is to produce estimates i v of i V , 1, , i m = . If the function H has additional parameters, we need to estimate those as well. Assume without loss of generality 1 0 m i i V = = ∑ and define ij i j V V δ = − . Estimation proceeds by tallying ij α , the number of times object i A is preferred to object j A after ij n comparisons. A sample estimate of ij π is / ij ij ij p n α = . We define ( ) ij ij H d p = and compute merit or scale estimates i v by ij i j d v v = − , i j ≠ , , 1, , i j m = . It can be shown that a least squares estimate of i V is 1 m i ij i j v d m ≠ = ∑ (3) This estimate holds regardless of H and is the usual method for Thurstone’s Case V model. Assume that each pair is observed a fixed (but possibly unequal) number of times. That is, the sums ij n are fixed and the tallies ij α are binomial random variables: ( ) (1 ) , 0,1, , ij ij ij ij ij ij ij ij ij ij n n P n α α α π π α α − = − =       (4) Owing to independence, the likelihood function is