Modeling Annotator Rationales with Application to Pneumonia Classification

We present a technique to leverage annotator rationale annotations for ventilator assisted pneumonia (VAP) classification. Given an annotated training corpus of 1344 narrative chest X-ray reports, we report results for two supervised classification tasks: Critical Pulmonary Infection Score (CPIS) and the likelihood of Pneumonia (PNA). For both tasks, our training data contain annotator rationale snippets (i.e., spans of text that are relevant to annotator decisions). Because we assume that the snippet is not marked in the test data, we first built a sequential labeler to detect the location of snippets. The detected snippets are then used by the CPIS and PNA classifiers. Our experiments demonstrate that having access to detected annotator rationale leads to an incremental improvement in classification accuracy from 0.858 to 0.871 for CPIS, and from 0.785 to 0.821 for PNA.