Arbeitspapiere der GMD 741 March 1993 GMD – Forschungszentrum

Multimodal interaction is employed in a variety of contemporary information systems in order to enhance the flexibility and naturalness of the user interface. In this paper, we propose a comprehensive framework which allows the modeling and specification of multimodal interactions. To this end, we employ an extended notion of ‘dialogue acts’ which can be performed by linguistic or non-linguistic means. On this basis we discuss the temporal structure of multimodal interaction. First, a comprehensive set of constraints is presented that describes all patterns of exchange which can be encountered during a cooperative information-seeking dialogue. Second, we introduce a strategic level of description, which allows the specification of the topical structure of the dialogue according to a selected information-seeking strategy. The model was used to design and implement the MERIT system (Multimedia Extensions to Retrieval Interaction Tools), and led to a reduction in the complexity of the user interface while preserving most of the useful, but sometimes confusing, dialogue options of advanced direct manipulative interfaces. 4 A Conversational Model of Multimodal Interaction Stein, Thiel