Pragmatic assessment and consistency in rating are among the subject matters which are still in need of more profound investigations. The importance of the issue is highlighted when remembering that inconsistency in ratings would surely damage the test fairness issue in assessment and lead to much diversity in ratings. Our principal concern in this study was observing the criteria that American...

In some online interactions, people use avatars to represent themselves and judge whether interaction partners should be trusted. However, little is known about human accuracy in perceptions of avatar trustworthiness. We conducted a two-stage study investigate are able accurately trustworthiness avatars. Stage 1, participants created using avatarmaker.com made decisions as trustees an incentivi...

Agreement between physicians in their classification of items such as mammograms for the presence of disease is an important tool in assessing the reliability of a diagnostic procedure, and the modeling of agreement data is a popular topic in the biomedical and social sciences. Interest often lies in assessing agreement in the underlying diagnostic procedure and making inferences for the popula...

UNLABELLED BACKGROUND The clinical global impression of severity (CGI-S) scale is a frequently used rating instrument for the assessment of global severity of illness in Central Nervous System (CNS) trials. Although scoring guidelines have been proposed to anchor these scores, the collection of sufficient documentation to support the derived score is not part of any standardized interview pr...

BACKGROUND A reliable and accurate estimation of liver size by physical examination is an important aspect of the clinical assessment of a patient. The scratch test uses auscultation to detect the lower liver edge by using the difference in sound transmission through the abdominal cavity over solid and hollow organs. The test is thought to be particularly useful if the abdomen is tense, distend...

The Kappa coefficient is widely used in assessing categorical agreement between two raters or two methods. It can also be extended to more than two raters (methods). When using Kappa, the shortcomings of this coefficient should be not neglected. Bias and prevalence effects lead to paradoxes of Kappa. These problems can be avoided by using some other indexes together, but the solutions of the Ka...

Nowadays crowdsourcing is widely used in supervised machine learning to facilitate the collection of ratings for unlabelled training sets. In order to get good quality results it is worth rejecting results from noisy/unreliable raters, as soon as they are discovered. Many techniques for filtering unreliable raters rely on the presentation of training instances to the raters identified as most a...

OBJECTIVE Interrater agreement and construct validity of the Revised Knox Preschool Play Scale (RKPPS) were examined. METHOD Two separately trained raters evaluated 38 typically developing children, ages 36 to 72 months. For each child, the raters observed two 15-min free-play sessions. RESULTS For the overall play age, the scores of the two raters were within 8 months of each other 86.8% o...