Dialog Act Modeling for Automatic Tagging and Recognition of Conversational Speech

We describe a statistical approach for modeling dialog acts in conversational speech, i.e., speechact-like units such as Statement, Question, Backchannel, Agreement, Disagreement, and Apology. Our model detects and predicts dialog acts based on lexical, collocational, and prosodic cues, as well as on the discourse coherence of the dialog act sequence. The dialog model is based on treating the discourse structure of a conversation as a hidden Markov model and the individual dialog acts as observations emanating from the model states. Constraints on the likely sequence of dialog acts are modeled via a dialog act N-gram. The statistical dialog grammar is combined with word N-grams, decision trees, and neural networks modeling the idiosyncratic lexical and prosodic manifestations of each dialog act. We develop a probabilistic integration of speech recognition with dialog modeling, to improve both speech recognition and dialog act classification accuracy. Models are trained and evaluated using a large hand-labeled database of 1155 conversations from the Switchboard corpus of spontaneous human-to-human telephone speech. We achieved good dialog act labeling accuracy (65% based on errorful, automatically recognized words and prosody, and 71% based on word transcripts, compared to a chance baseline accuracy of 35% and human accuracy of 84%) and a small reduction in word recognition error.