<scp>FaithDial</scp>: A Faithful Benchmark for Information-Seeking Dialogue

Abstract The goal of information-seeking dialogue is to respond seeker queries with natural language utterances that are grounded on knowledge sources. However, systems often produce unsupported utterances, a phenomenon known as hallucination. To mitigate this behavior, we adopt data-centric solution and create FaithDial, new benchmark for hallucination-free dialogues, by editing hallucinated responses in the Wizard Wikipedia (WoW) benchmark. We observe FaithDial more faithful than WoW while also maintaining engaging conversations. show can serve training signal for: i) hallucination critic, which discriminates whether an utterance or not, boosts performance 12.8 F1 score BEGIN compared existing datasets coherence; ii) high-quality generation. series state-of-the-art models propose auxiliary contrastive objective achieves highest level faithfulness abstractiveness based several automated metrics. Further, find benefits generalize zero-shot transfer other datasets, such CMU-Dog TopicalChat. Finally, human evaluation reveals generated trained perceived interpretable, cooperative, engaging.