Multimodal conversational dialogue systems consisting of numerous software components create challenges for the underlying software architecture and development practices. Typically, such systems are built on separate, often preexisting components developed by different organizations and integrated in a highly iterative way. The traditional dialogue system pipeline is not flexible enough to add...

In this position paper, we present an approach geared toward rapid prototyping of multimodal interfaces with SMUIML (Synchronized Multimodal User Interfaces Markup Language) and HephaisTK toolkit. The goal of SMUIML and HephaisTK is to offer developers with: on one side, a language allowing clear description of human-machine multimodal dialog and control over the way multiple input modalities h...

Multimodal interaction allows users to specify commands using combinations of inputs from multiple different modalities. For example, in a local search application, a user might say “gas stations” while simultaneously tracing a route on a touchscreen display. In this demonstration, we describe the extension of our cloud-based speech recognition architecture to a Multimodal Semantic Interpretati...

Human-to-human conversation remains such a significant part of our working activities because of its naturalness. Multimodal interaction systems combine visual information with voice, gestures, and other modalities to provide flexible and powerful dialogue approaches. The use of integrated multiple input modes enables users to benefit from the natural approach used in human communication. In th...

This paper discusses software architectures of multimodal systems. The recent availability of new input technologies brought a whole new type of systems, able to support communication with the user through multiple interaction channels. Multimodal systems that allow modalities to be combined seem to be the most promising in the field of multimodal interaction. The aim is to overcome limitations...

|This paper presents a statistical approach to developing multimodal recognition systems and, in particular, to integrating the posterior probabilities of parallel input signals involved in the multimodal system. We rst identify the primary factors that innuence multimodal recognition performance by evaluating the multimodal recognition probabilities. We then develop two techniques, an estimate...

Multimodal systems support people with different needs and different features during the interaction process making it more easy and natural. However naturalness can usually produce ambiguous interpretation. This paper discusses ambiguities in multimodal interaction proposing in particular two different examples of ambiguities: a) multimodal ambiguities produced combining un-ambiguous modal inf...

As pervasive computing is widely available during daily activities, wearable input devices which promote an eyes-free interaction are needed for easy access and safety. We propose a textile wearable device which enables a multimodal sensing input for an eyes-free mobile interaction during daily activities. Although existing input devices possess multimodal sensing capabilities with a small form...