This study focuses on the comparison of expert assessment of virtual surgical procedures through Objective Structured Assessment of Technical Skills (OSATS) with the automatic assessment and feedback provided by a surgical simulator for hysteroscopic procedures. The existing multi-metric scoring system of the simulator was extended to include hysteroscopic myomectomy. The original OSATS was als...

Currently 19%-28% of Internet users participate in online health discussions. In this work, we study sentiments expressed on online medical forums. As well as considering the predominant sentiments expressed in individual posts, we analyze sequences of sentiments in online discussions. Individual posts are classified into one of the five categories encouragement, gratitude, confusion, facts, an...

In assessment centers (ACs), research on eliciting candidate behavior and evaluating candidate behavior have largely followed independent paths. This study integrates trait activation and trait rating models to posit hypotheses about the effects of behavior elicitation via situational cues on key assessor observation and rating variables. To test the hypotheses, a series of experimental and fie...

We discuss and evaluate a new annotation scheme and discourse-analytic method, the QUD-tree framework. We present an annotation study, in which the framework, based on the concept of Questions under Discussion, is applied to English and German interview data, using TreeAnno, an annotation tool specially developed for this new kind of discourse annotation. The results of an inter-annotator agree...

This paper describes metaTED – a freely available corpus of metadiscursive acts in spoken language collected via crowdsourcing. Metadiscursive acts were annotated on a set of 180 randomly chosen TED talks in English, spanning over different speakers and topics. The taxonomy used for annotation is composed of 16 categories, adapted from Ädel (2010). This adaptation takes into account both the ma...

Patients are often exposed to medical terms, such as anosognosia, myelodysplastic, or hepatojejunostomy, that can be semantically complex and hardly understandable by non-experts in medicine. Hence, it is important to assess which words are potentially non-understandable and require further explanations. The purpose of our work is to build specific lexicon in which the words are rated according...

Weighted kappa is a widely used statistic for summarizing inter-rater agreement on a categorical scale. For rating scales with three categories, there are seven versions of weighted kappa. It is shown analytically how these weighted kappas are related. Several conditional equalities and inequalities between the weighted kappas are derived.The analytical analysis indicates that the weighted kapp...

This paper describes a system for interannotator agreement analysis of ERE annotation, focusing on entity mentions and how the higher-order annotations such as EVENTS are dependent on those entity mentions. The goal of this approach is to provide both (1) quantitative scores for the various levels of annotation, and (2) information about the types of annotation inconsistencies that might exist....

One of the primary barriers to the acceptance of surgical simulators is that most simulators still require a significant amount of an instructing surgeon's time to evaluate and provide feedback to the students using them. Thus, an important area of research in this field is the development of metrics that can enable a simulator to be an essentially self-contained teaching tool, capable of ident...

The level of agreement among participants is a key aspect of gesture elicitation studies, and it is typically quantified by means of agreement rates (AR). We show that this measure is problematic, as it does not account for chance agreement. The problem of chance agreement has been extensively discussed in a range of scientific fields in the context of inter-rater reliability studies. We review...