Predicting the end of user input turns in a multimodal system can be complex. User interactions vary across a spectrum from single, unimodal inputs to multimodal combinations delivered either simultaneously or sequentially. Early multimodal systems used a fixed duration temporal threshold to determine how long to wait for the next input before processing and integration. Several recent studies ...

Predicting the end of user input turns in a multimodal system can be complex. User interactions vary across a spectrum from single, unimodal inputs to multimodal combinations delivered either simultaneously or sequentially. Early multimodal systems used a fixed duration temporal threshold to determine how long to wait for the next input before processing and integration. Several recent studies ...

In order to realize their full potential, multimodal interfaces need to support not just input from multiple modes, but single commands optimally distributed across the available input modes. A multimodal language processing architecture is needed to integrate semantic content from the different modes. Johnston 1998a proposes a modular approach to multimodal language processing in which spoken ...

The present study addresses two questions: On a non-visual multimodal interface for textual information browsing, (1) how prevalent is input modality switching as an error handling strategy, and (2) how much does an input modality need to fail before input modality switching occurs. The results indicate that although switching input modalities to correct errors is an expected practice on multim...

This Paper describes the approach of multimodal human-machine interaction. It presents a definition and implementation of an interface framework, which allows for processing multiple instances of input. In a study on the combination of eye movements and brain signals as input, the use of multimodal interaction is proven.

Ever since Bolt’s seminal paper, ”Put that there: Voice and Gesture at the Graphics Interface”, the notion that multiple modes of input could be used to interact with computer applications has been an active area of human computer interaction research (Bolt 1980). This combiniation of different forms of input (e.g., speech, gesture, touch, eye gaze) is known as multimodal interaction and its go...

This paper presents a knowledge-based method of early-stage high level multimodal fusion of data obtained from speech input and visual scene. The ultimate goal is to develop a human-computer multimodal interface to assist elderly people living alone at home to perform their daily activities, and to support their active ageing and social cohesion. Crucial for multimodal high level fusion and suc...

Multimodal Interaction has the potential to significantly increase the ease of use of human computer interaction (HCI). At the same time, due to error-prone recognition based inputs, it is merely used in real-world applications. While literature on multimodal input fusion describes modeling of different user behaviors as a key for increased robustness, it still failed to prove it’s practical us...

Virtual Environments (VE) represent advanced human computer interaction systems, communicating over several channels of information. The basic terms we need to describe human VE interaction are “communication channel” and “multimodality”. The communication channel is as a pathway between the user and the simulation that mediates the interaction. Multimodality is the quality of a system, which a...