Obtaining Predictive Results with an Objective Evaluation of Spoken Dialogue Systems: Experiments with the DCR Assessment Paradigm
نویسندگان
چکیده
The DCR methodology is a framework that proposes a generic and detailed evaluation of spoken dialog systems. We have already detailed (Antoine et al., 1998) the theoretical bases of this paradigm. In this paper, we present some experimental results on spoken language understanding that show the feasibility and the reliability of the DCR evaluation as well as its ability to provide a detailed diagnosis of the system’s behaviour. Finally, we highlight the extension of the DCR methodology to dialogue management.
منابع مشابه
Predicting Dialogue Acts from Prosodic Information
In this paper, the influence of intonation to recognize dialogue acts from speech is assessed. Assessment is based on an empirical approach: manually tagged data from a spoken-dialogue and video corpus are used in a CARTstyle machine learning algorithm to produce a predictive model. Our approach involves two general stages: the tagging task, and the development of machine learning experiments. ...
متن کاملToward an objective and generic Method for Spoken Language Understanding Systems Evaluation: an extension of the DCR method
In this paper, we present an extension of the DCR method, which is a framework for the deep evaluation of Spoken Language Understanding (SLU) Systems. The key point of our contribution is the use of a linguistic typology in order to generate an evaluation corpus that covers a significant number of the linguistic phenomena we want to evaluate our system on. This allows to have more objective and...
متن کاملResponse Generation in Dialogue Using a Tailored PCFG Parser
This paper presents a parsing paradigm for natural language generation task, which learns a tailored probabilistic context-free grammar for encoding meaning representation (MR) and its corresponding natural language (NL) expression, then decodes and yields natural language sentences at the leaves of the optimal parsing tree for a target meaning representation. The major advantage of our method ...
متن کاملEvaluation for Darpa Communicator Spoken Dialogue Systems
The overall objective of the DARPA COMMUNICATOR project is to support rapid, cost-effective development of multi-modal speechenabled dialogue systems with advanced conversational capabilities, such as plan optimization, explanation and negotiation. In order to make this a reality, we need to find methods for evaluating the contribution of various techniques to the users’ willingness and ability...
متن کاملResults from an Evaluation of a Dialogue Manager
We give the results from an evaluation of the dialogue manager of a developing, Slovenian and Croatian spoken dialogue system for weather-information retrieval. The results are based on two Wizard-of-Oz experiments and the application of the PARADISE evaluation framework. The only difference between both experiment settings is in the dialogue-management manner, i.e., while in the first experime...
متن کامل