An agent-based architecture for multimodal interaction

In this paper, an executable generic process model is proposed for combined verbal and non-verbal communication processes and their interaction. The agent-based architecture can be used to create multimodal interaction. The generic process model has been designed, implemented and used to simulate di!erent types of interaction between verbal and non-verbal communication processes: a non-verbal communication process can add and modify content to a verbal communication process, but can also provide a protocol for the (control of the) verbal communication process. With respect to the communication protocol both stimulus}response behaviour and deliberative behaviour have been modelled and simulated. The semantics of the model has been formalized by threelevelled partial temporal models, covering both the material and mental proceses and their relations. ( 2001 Academic Press