This paper presents a brief survey of currently ongoing research at DIST – InfoMus Lab on multimodal interfaces with a particular focus on Tangible Acoustic Interfaces (TAIs). A relevant aspect of our research is the multimodal analysis of the high-level expressive, emotional information, the involved non-verbal communication mechanisms, the role of such information and mechanisms in the design...

This article presents a tripartite framework for analyzing multimodal texts. The three analytical perspectives presented include: (1) perceptual, (2) structural, and (3) ideological analytical processes. Using Anthony Browne’s picturebook Piggybook as an example, assertions are made regarding what each analytical perspective brings to the interpretation of multimodal texts and how these perspec...

— Human-to-human conversation remains such a significant part of our working activities because of its naturalness. Multimodal interaction systems combine visual information with voice, gestures and other modalities to provide flexible and powerful dialogue approaches. The use of integrated multiple input modes enables users to benefit from the natural approach used in human communication. Howe...

A genuine comprehension of the process of human communication within organisations requires a detailed understanding of pragmatics. Accordingly, our aim here is to work towards a richer means of analysing and describing multimodal documents at the pragmatic level than is currently available within the framework of Organisational Semiotics. Our basic strategy is to take six of the most widely kn...

We introduce TYCOON a framework we are developing for the analysis of human verbal and non-verbal behavior. This framework includes a typology made of six primitive types of cooperation between communicative modalities: equivalence, specialization, transfer, redundancy, complementarity and concurrency. We have used this typology when annotating videotaped multimodal humancomputer interaction an...

The Writing and Communication Program at Georgia Tech emphasizes three critical concepts: communication is rhetorical, communication involves a process, and communication is multimodal. You probably already know a considerable amount about these concepts, but English 1101 and English 1102 help you build on what you already know by encouraging you to explore your ideas about communication and to...

W. Wahlster, E. André, W. Finkler, H.-J. Profitlich and T. Rist, Plan-based integration of natural language and graphics generation, Artificial Intelligence 63 (1993) 387-427. Multimodal interfaces combining natural language and graphics take advantage of both the individual strength of each communication mode and the fact that several modes can be employed in parallel. The central claim of thi...

In multi-party interactions humans use available communication modes in predictable ways. For example, the dialogue theories of Conversational Implicature (Grice 1975) and Giveness Theory (Gundel, Hedberg et al. 1993) have both been applied successfully in the analysis of multi-party, multimodal settings (Chai, Prasov et al. 2005). At times people use multiple modes of communication in compleme...

In the recent years multimodal interfaces have acquired an important role in human computer interaction applications. Humans use multimodality to reduce ambiguity and incompleteness of information. Seemingly they are able to switch easily from one modality to the other and fuse the information from different multimodal sources. The goal of our research was to develop a crisis based human like m...

Emotion is assuming increasing importance in human computer interaction (HCI), in general, with the growing feeling that emotion is central to human communication and intelligence. Users expect not just functionality as a factor of usability, but experiences, matched to their expectations, emotional states, and interaction goals. Endowing computers with this kind of intelligence for HCI is a co...