Multimodal interfaces require effective parsing and understanding of utterances whose content is distributed across multiple input modes. Johnston 1998 presents an approach in which strategies for multimodal integration are stated declaratively using a unification-based grammar that is used by a multidimensional chart parser to compose inputs. This approach is highly expressive and supports a b...

Predicting the end of user input turns in a multimodal system can be complex. User interactions vary across a spectrum from single, unimodal inputs to multimodal combinations delivered either simultaneously or sequentially. Early multimodal systems used a fixed duration temporal threshold to determine how long to wait for the next input before processing and integration. Several recent studies ...

The increasing need to access information everywhere and at any time leads us to believe that future user interfaces, through which users interact with pervasive computing systems, must enable both device and modality independence. The pervasive computing paradigm sees almost every object in the everyday environment as a system able to communicate with users and other systems in their own langu...

In order to realize their full potential, multimodal interfaces need to support not just input from multiple modes, but single commands optimally distributed across the available input modes. A multimodal language processing architecture is needed to integrate semantic content from the different modes. Johnston 1998a proposes a modular approach to multimodal language processing in which spoken ...

In medical research and practice, imaging is now playing a crucial role. With rapidly developing molecular imaging, the role of in vivo imaging is expected to expand much more. In molecular imaging, one of the recent trends is multimodal imaging, which theoretically combines all the strengths of each imaging modality and consequently provides synergism. For this, many multimodal imaging probes ...

Multimodal conversational interfaces provide a natural means for users to communicate with computer systems through multiple modalities such as speech and gesture. To build effective multimodal interfaces, automated interpretation of user multimodal inputs is important. Inspired by the previous investigation on cognitive status in multimodal human machine interaction, we have developed a greedy...

With the success of multimedia and mobile devices, humancomputer interfaces combining several communication modalities such as speech and gesture may lead to more "natural" humancomputer interaction. Yet, developing multimodal interfaces requires an understanding (and thus the observation and analysis) of human multimodal behavior. In the field of annotation of multimodal corpus, there is no st...

INTRODUCTION Multimodal rehabilitation programmes enhance recovery and may reduce the postoperative complication rate and hospital stay after elective colorectal surgery. PATIENTS AND METHOD A clinical pathway of multimodal rehabilitation in elective colorectal surgery was developed. After implementation, 90 consecutive patients received multimodal rehabilitation care from April to December 2...

ZHANG, WEI. Multimodal Pedagogical Planning for 3D Learning Environments A Unified Framework. (Under the Direction of Dr. James C. Lester and Dr. R. Michael Young.) Pedagogical planning lies at the heart of knowledge-based learning environments. In recent years, multimodality and authoring have become key issues in the creating of learning environments. The purpose of this research has been to ...

We propose an architecture of neural network that can learn and integrate sequential multimodal information using Long Short Term Memory. Our model consists of encoder and decoder LSTMs and multimodal autoencoder. For integrating sequential multimodal information, firstly, the encoder LSTM encodes a sequential input to a fixed range feature vector for each modality. Secondly, the multimodal aut...