Interactive spaces provide new affordance, allowing real-time interaction between users and digital/physical space, the possibilities of multimodal, whole body interaction and working with new design methods or input devices. The challenge is however that interactive spaces are not easily modifiable for different purposes. As every interactive space is a unique place, software must be adapted t...

Most of the existing multimodal prototypes enabling users to combine 2D gestures and speech are task-oriented. They help adult users to solve particular information tasks often in 2D standard Graphical User Interfaces. This paper describes the NICE HCA system which aims at demonstrating multimodal conversation between humans and embodied historical and literary characters. The target users are ...

The design and implementation of hybrid symbolic/statistical architectures is a major area of interest in current multimodal system development. Such an architecture attempts to improve multimodal recognition and disambiguation rates by using corpus-based statistics to weight the contributions from various input streams. This is in contrast to current architectures that assume independence betw...

In this paper we describe the computational and architectural requirements for systems which support real-time multimodal interaction with an embodied conversational character. We argue that the three primary design drivers are real-time multithreaded entrainment, processing of both interactional and propositional information, and an approach based on a functional understanding of human face-to...

This paper presents a statistical approach to developing multimodal recognition systems and, in particular, to integrating the posterior probabilities of parallel input signals involved in the multimodal system. We first identify the primary factors that influence multimodal recognition performance by evaluating the multimodal recognition probabilities. We then develop two techniques, an estima...

Modern user interfaces can take advantage of multiple input modalities such as speech, gestures, handwriting... to increase robustness and flexibility. The construction of such multimodal interfaces would be greatly facilitated by a unified framework that provides methods to characterize and interpret multimodal inputs. In this paper we describe a semantic model and a multimodal grammar structu...

The design and implementation of hybrid symbolic/statistical architectures is a major area of interest in current multimodal system development. Such an architecture attempts to improve multimodal recognition and disambiguation rates by using corpus-based statistics to weight the contributions from various input streams. This is in contrast to current architectures that assume independence betw...

This paper introduces a method that generates simulated multimodal input to be used in testing multimodal system implementations, as well as to build statistically motivated multimodal integration modules. The generation of such data is inspired by the fact that true multimodal data, recorded from real usage scenarios, is difficult and costly to obtain in large amounts. On the other hand, thank...

This paper discusses multimodal systems. We have defined a taxonomy of multimodal interactions and built two example multimodal systems. We consider several levels where the fusion can take place in the software architecture of those systems. We emphasize the need for a common abstract representation to represent input data and commands, and to combine them successfully. This is still an area o...

The PeopleFinder is a knowledge-based tool to assist users in determining the whereabouts of other staff located in an office or network environment. The tool makes use of several modes of input and output, as well as employing a number of interface and communications media with which to present information and interconnect geographically distributed system users. The accompanying video contain...