This paper describes the structure of a test suite for evaluation of clinical question answering systems; presents several manually compiled resources found useful for test suite generation; and describes the adaptation of these resources for evaluation of a clinical question answering system.

During the recent Dialog State Tracking Challenge (DSTC), a fundamental question was raised: “Would better performance in dialog state tracking translate to better performance of the optimized policy by reinforcement learning?” Also, during the challenge system evaluation, another nontrivial question arose: “Which evaluation metric and schedule would best predict improvement in overall dialog p...

This paper presents the evaluation of a Temporal QA system for the treatment of temporal complex questions. The system was implemented as a multilayered architecture where complex temporal questions are first decomposed into simple questions, according to the temporal relations expressed in the original question. These simple questions are then processed independently by our standard Question A...

This paper describes the Chinese question answering system DLUT for the CS-CS Subtask evaluation of NTCIR-8 CCLQA. The system utilizes Situation Unit (SU), combined syntactic and semantic information, as a basic processed unit to get candidate answers, wherein the SUs in the question sentences are matched with those in the texts of corpus. In this evaluation, answers are presented in the form o...