Human face-to-face communication is a complex multimodal signal. We use words (language modality), gestures (vision modality) and changes in tone (acoustic modality) to convey our intentions. Humans easily process and understand face-toface communication, however, comprehending this form of communication remains a significant challenge for Artificial Intelligence (AI). AI must understand each m...

Human communication is naturally multimodal, and substantial focus has examined the semantic correspondences in speech-gesture and text-image relationships. However, visual narratives, like those in comics, provide an interesting challenge to multimodal communication because the words and/or images can guide the overall meaning, and both modalities can appear in complicated "grammatical" sequen...

For many application areas, where a task is most naturally represented by talking or where standard input devices are difficult to use or not available at all, virtual characters can be well suited as an intuitive man-machineinterface due to their inherent ability to simulate verbal as well as nonverbal communicative behavior. This type of interface is made possible with the help of multimodal ...

This paper describes articulatory adaptation in humanto-human communication, a computational model of such adaption by human communicators, and the application of this work — intended primarily for the design of communication aids, but with potential application to multimodal communicative agents.

Multimodal data of the complex human anatomy contain a wealth of information. To visualize and explore such data, techniques for emphasizing important structures and controlling visibility are essential. Such fused overview visualizations guide physicians to suspicious regions to be analyzed in detail, e.g. with slice-based viewing. We give an overview of state of the art in multimodal medical ...

UI on the Fly is a system that dynamically presents coordinated multimodal content through natural language and a small-screen graphical user interface. It adapts to the user’s preferences and situation. Multimodal Functional Unification Grammar (MUG) is a unification-based formalism that uses rules to generate content that is coordinated across several communication modes. Faithful variants ar...

With the success of multimedia and mobile devices, humancomputer interfaces combining several communication modalities such as speech and gesture may lead to more "natural" humancomputer interaction. Yet, developing multimodal interfaces requires an understanding (and thus the observation and analysis) of human multimodal behavior. In the field of annotation of multimodal corpus, there is no st...

this paper presented a multimodal discourse analysis of some visual images in the political rally discourse of 2011 electioneering campaigns in southwestern nigeria. the data comprised purposively selected political visual artefacts from political rallies across the six southwestern states in nigeria (osun, oyo, ondo, ekiti, ogun, and lagos). the data were analyzed using halliday’s (1985) syste...

The purpose of this dissertation is to propose and investigate multimodal expression of personified media, with multiple modalities, focusing on the expressive strength of voice and gesture. The results are expected to be used in a practical construction of personified representation. There are many of research works on and products of, personified media such as educational puppets, humanoid ro...