Effect of Linguistic Contents on Human Estimation of Internal State of Dialog System Users

We have studied estimation of dialog system users’ internal state before the input utterance. In a practical use of a dialogue-based system, a user is often perplexed with the prompt. An ordinary system provides more detailed information to the user taking time to input, but the help is meddlesome for the user considering the answer to the prompt. To make an appropriate response, the spoken dialogue system needs to consider the user’s internal state before the user’s input. In the previous paper, we proposed a method for estimating the internal state using multi-modal cues; however, we did not separate effects of several factors (e.g. linguistic information of the prompt, visual information, and acoustic information). Thus, it was not clear which factor affects the evaluation of the dialog session to what extent. In this paper, we examined more detailed evaluation by human evaluators, separating linguistic contents (the system’s prompt utterance and the user’s reply utterance) from the other non-verbal behavior, and assessed the effect of the linguistic contents on the estimation of the user’s internal state.