Multimodal conversational interfaces provide a natural means for users to communicate with computer systems through multiple modalities such as speech and gesture. To build effective multimodal interfaces, understanding user multimodal inputs is important. Previous studies have shown that user language behavior does not occur randomly, but rather follows certain linguistic and cognitive princip...

A robot that interacts with a human has to be able to interpret information from various input channels: it needs to understand and analyse the utterances by the human, it has to keep track of its own environment using sensors, and it needs to incorporate background knowledge about the task it was built for. Typically, a human-robot interaction system has various specialised system components t...

The growing interest in multimodal interface design is inspired in large part by the goals of supporting more transparent, flexible, efficient, and powerfully expressive means of human–computer interaction than in the past. Multimodal interfaces are expected to support a wider range of diverse applications, be usable by a broader spectrum of the average population, and function more reliably un...

This contribution focuses on multimodal interaction techniques for a mobile communication and assistance system on a robot platform. The system comprises of acoustic, visual and haptic input modalities. Feedback is given to the user by a graphical user interface and a speech synthesis system. By this, multimodal and natural communication with the robot system is possible.

The purpose of this thesis is to address the fact that users’ input to training systems in virtual environments is not suited to their natural skills and abilities. These skills and abilities include speaking and gesturing with their bodies. This mismatch may have negative effects on their usage of the virtual environment. One assumption guiding this thesis is that it would increase immersion t...

A design methodology for multimodal-controlled application has been developed using Wizard-of-Oz simulations as the principal mechanism for evaluating and getting input for dialogue design. This methodology may enable multimodal application developers to support dialogues that are optimal with respect to naturalness, especially on a pragmatic level, given the technical restrictions.

The performance prediction and task sequencing in traditional adaptive intelligent tutoring systems needs information gained from expert and domain knowledge. In a former work a new efficient task sequencer based on a performance prediction system was presented, which only needs former performance information but not the expensive expert and domain knowledge. In this paper we aim to support thi...

Much research has been directed towards developing mul-timodal interfaces in the past twenty years. Many current multimodal systems, however, can only handle multimodal inputs at the sentence level. The move towards multimodal dialogue signiicantly increases the complexity of the system as the representations of the input now range over time and input modality. We developed a framework to addre...

Multimodal conversational interfaces provide a natural means for users to communicate with computer systems through multiple modalities such as speech and gesture. To build effective multimodal interfaces, automated interpretation of user multimodal inputs is important. Inspired by the previous investigation on cognitive status in multimodal human machine interaction, we have developed a greedy...