Integrating Pointing Gestures into a Spanish-spoken Dialog System for Conversational Service Robots

In this paper we present our work on the integration of human pointing gestures into a spoken dialog system in Spanish for conversational service robots. The dialog system is composed by a dialog manager, an interpreter that guides the spoken dialog and robot actions, in terms of user intentions and relevant environment stimuli associated to the current conversational situation. We demonstrate our approach by developing a tour–guide robot that is able to move around its environment, visually recognize informational posters, and explain sections of the poster selected by the user via pointing gestures. This robot also incorporates simple methods to qualify confidence in its visual outcomes, to inform about its internal state, and to start error–prevention dialogs whenever necessary. Our results show the reliability of the overall approach to model complex multimodal human–robot interactions.