With the abundance of multimedia in web databases and the increasing user need for content of many modalities, such as images, sounds, etc., new methods for retrieval and visualization of multimodal media are required. In this paper, novel techniques for retrieval and visualization of multimodal data, i.e. documents consisting of many modalities, are proposed. A novel cross-modal retrieval fram...

Communication between people is inherently multimodal. People employ speech, facial expressions, eye gaze, and gesture, among other facilities, to support communication and cooperative activity. Complexity of communication increases when a person is without a modality such as hearing, often resulting in dependence on another person or an assistive device to facilitate communication. This paper ...

This research investigates the use of multimodal metaphors to communicate information in the interface of e-learning application in order to reduce the visual communication by incorporating auditory stimuli. Three experimental studies conducted to investigate the effect of using multimodal metaphors in e-learning applications. The first experiment introduced an empirical study to investigate th...

The paper presents an architecture for multimodal mood-based annotation systems. The architecture aims at the implementation of interactive multimodal systems to support communities of users in the creation and management of annotations in locative media projects. The annotations are multimodal in that they can be created and accessed through visual and audio interaction. The annotations are mo...

This article proposes a methodology to create a multimodal corpus that can be shared with a group of researchers in order to analyze synchronous online pedagogical interactions. Epistemological aspects involved in studying online interactions from a multimodal and semiotic perspective are addressed. Then, issues and challenges raised by corpus creation and sharing are examined with a particular...

-This paper introduces an empirical study to investigate the use of multimodal metaphors in e-learning application in particular within the area of note-taking. The aim of the study is to measure and compare the level of usability of these metaphors. In order to carry out this comparative investigation, two experimental interfaces were built and performed by forty users. In the first interface ...

In this work, a classifier combination approach for computer aided diagnosis (CADx) of breast mass lesions in mammography (MG) and magnetic resonance imaging (MRI) is investigated, using a database with 278 and 243 findings in MG resp. MRI including 98 multimodal (MM) lesion annotations. For each modality, feature selection was performed separately with linear Support Vector Machines (SVM). Usi...

With the increase of multimedia content on the WWW, multimodal presentations using interactive lifelike agents become an attra ctive style to deliver information. However, for many people it is not easy to write multimodal presentations. This is because of the complexity of describing various behaviors of character agents based on a particular character system with individual (often low-level) ...

This paper describes how the thermal user interface system can be integrated into other input devices to develop multimodal user interfaces that present multisensory feedback. Multimodal virtual environments supporting both haptic force and haptic thermal feedback will allow researchers to investigate a wide range of research issues involved in presence study. Among presence-related research qu...

Multimodal neuroimaging has become a mainstay of basic and cognitive neuroscience in humans and animals, despite challenges to consider when acquiring and combining non-redundant imaging data. Multimodal data integration can yield important insights into brain processes and structures in addition to spatiotemporal resolution complementarity, including: a comprehensive physiological view on brai...