KPT: Keyword-Guided Pre-training for Grounded Dialog Generation

Incorporating external knowledge into the response generation process is essential to building more helpful and reliable dialog agents. However, collecting knowledge-grounded conversations often costly, calling for a better pre-trained model grounded that generalizes well w.r.t. different types of knowledge. In this work, we propose KPT (Keyword-guided Pre-Training), novel self-supervised pre-training method without relying on extra annotation. Specifically, use language extract most uncertain tokens in as keywords. With these keywords, construct two kinds pre-train model, aiming at handling scenarios: (1) should be faithfully grounded; (2) it can selectively used. For former, grounding consists keywords extracted from response. latter, additionally augmented with other utterances same dialog. Since itself, easily performed large volume variety dialogue data. We considered three data sources (open-domain, task-oriented, conversational QA) total 2.5M dialogues. conduct extensive experiments various few-shot tasks, including acts, graphs, persona descriptions, Wikipedia passages. Our comprehensive analyses demonstrate consistently outperforms state-of-the-art methods tasks diverse