Crowdsourcing for Spoken Dialog Systems Evaluation

A spoken dialog system (SDS) is a computer system which supports human-computer conversations in specific knowledge domains. It integrates technologies including speech recognition, natural language understanding, dialog modeling, language generation, and textto-speech synthesis. Advances in speech and language technologies have made SDS an important research area and have brought about systems in a wide variety of applications, such as flight information (Hirschman et al. (1993)), bus schedule inquiries (Raux et al. (2005)), stock market information delivery (Meng et al. (2004a)), tourist information (Wu et al. (2006)) and student tutoring (Litman et al. (2006)). To facilitate the development of SDS and compare the performance of different systems, it is necessary to conduct SDS evaluation with appropriate methodologies. A typical SDS architecture is illustrated in Figure 1.1. This implements a dialog interaction between two interlocutors, which consists of a series of dialog turns. A spoken dialog turn is a process in which one participant A utters something to the other B, and B interprets A’s utterance and then responds accordingly. As illustrated in Figure 1.1, the first step for the system is to recognize the user’s speech with automatic speech recognition and interpret the underlying meaning with natural language understanding technologies. This involves extracting the user’s communicative (and informational) goal and inferring the appropriate follow up actions and responses. Language understanding involves a variety of methods, such as the use of parsers and grammars (Seneff (1992); Ward and Issar (1994)), belief networks (Meng et al. (1999, 2004b)), etc. The dialog model is the principal component of a dialog system which maintains the history of the dialog, decides which action is appropriate based on language understanding, and controls the dialog flow. A dialog model typically incorporates dialog states, state transitions, and a dialog policy. A dialog state represent the results of performing system actions in previous states; state transitions allow dialogs to move forward; and the dialog policy determines how