This paper presents a part of an on-going project that aims at developing an instrument to evaluate English language teaching textbooks, called the English Language Teaching (ELT) Textbook Evaluation Checklist. The present study sought to evaluate the validity, reliability and economy of the developed checklist. It also aimed to find out to what extent the users of the checklist were satisfied ...

BACKGROUND There have been several attempts to systematically assess performance in bronchoscopy. Earlier validation studies have used bronchoscopy simulators, not real-life performance in patients. OBJECTIVES The aim of this study was to explore the reliability and validity of an assessment tool aimed for the use in a clinical setting. METHODS Five junior residents, 5 senior residents and ...

This paper presents the design and results of a crowdsourcing experiment on the recognition of Italian event nominals. The aim of the experiment was to assess the feasibility of crowdsourcing methods for a complex semantic task such as distinguishing the eventive interpretation of polysemous nominals taking into consideration various types of syntagmatic cues. Details on the theoretical backgro...

Despite a great deal of research effort, disfluency and laughter annotation is still an unsolved problem, both in terms of consensus for a general applicable system, and in terms of annotation agreement metrics. In this paper we present a new annotation scheme within a light-weight mark-up for spontaneous speech. We show, despite the low overhead required for understanding the annotation protoc...

A modified version of the Scale for Suicidal Ideation was developed for use by paraprofessionals. The modifications included (a) addition of standardized prompt questions, (b) a standardized sequence of administration, (c) modification of the rating points to increase their specificity and range, (d) development of initial screening scores, and (e) selection of items for inclusion in the scale ...

There is a need for a short instrument to assess the multiple areas of dysfunction as well as drinking dyscontrol in alcoholics. The Brief Addiction Rating Scale (BARS), covering 10 areas of functioning has been developed to meet this need. This report describes the rationale and development of this new scale. A high interrater reliability (0.9) on all the items of the scale and the ease of adm...

The Thurstone Word Fluency Test (TWFT) is a widely used neuropsychological instrument. However, data regarding its psychometric properties are lacking. The results of the present study suggest that the TWFT possesses excellent test-retest and inter-rater reliability, in addition to good construct validity. However, its criterion validity is limited by its lack of specificity and sensitivity. Th...

This paper presents an annotation scheme for modelling citation contexts in scientific articles. We present an argumentation framework based on the Toulmin model for scientific articles and develop an annotation scheme with different context types based on the argumentation model. We present the results of the inter-rater reliability study carried out for studying the reliability of our annotat...

PURPOSE To evaluate inter- and intra-rater reliability for the interpretation of MTI Photoscreener photographs taken in a population of Native American preschool children with a high prevalence of astigmatism. METHODS Photographs of 369 children were rated by 11 nonexpert and 3 expert raters. Photographs for each child were scored as pass, refer, or retake. Nonexpert raters scored photos on t...

Objective Several lower limb tendinopathy treatment modalities involve identification of pathological paratendinous or intratendinous neovascularisation to target proposed co-location of painful neoneuralisation. The ability to reliably locate and assess the degree of neovascularity is therefore clinically important. The Modified Ohberg Score (MOS) is frequently used to determine degree of neov...