Specification and interpretation of multimodal dialogue models for human-robot interaction

In this paper a conceptual model for the design and construction of interactive multimodal systems is presented. This model is based on a representational language for the specification of dialogue models and its associated program interpreter. Dialogues models are domain and modality independent conversational schemes characterizing the dynamic of a multimodal interaction, and domain specific applications are represented as sets of dialogue models. Dialogue models are specified in terms of the intentions and actions, expressed and performed, in conversation situations, and represent the context for multimodal interpretation. In this paper, the language for representing dialogue models and the program interpreter are described in detail. The model is implemented as a multi-agent environment in which there is a main agent embedding the dialogue model’s interpreter, the dialogue manager, and a specialized agent for each input and output modality. The theory has been tested with a simple application supporting spoken natural language input and output, graphics and video output, and also the motor action of a robot, which is also considered as an output modality. The present model has been implemented in the robot Golem, which has been widely demonstrated in Mexico, with a very positive response.