Continuous Measurement Scales in Human Evaluation of Machine Translation

We explore the use of continuous rating scales for human evaluation in the context of machine translation evaluation, comparing two assessor-intrinsic qualitycontrol techniques that do not rely on agreement with expert judgments. Experiments employing Amazon’s Mechanical Turk service show that quality-control techniques made possible by the use of the continuous scale show dramatic improvements to intra-annotator agreement of up to +0.101 in the kappa coefficient, with inter-annotator agreement increasing by up to +0.144 when additional standardization of scores is applied.