We propose a Deep Belief Network architecture for learning a joint representation of multimodal data. The model defines a probability distribution over the space of multimodal inputs and allows sampling from the conditional distributions over each data modality. This makes it possible for the model to create a multimodal representation even when some data modalities are missing. Our experimenta...

In this paper we present a proposal for extending the standard Wizard of Oz experimental methodology to language-enabled multimodal systems. We first discuss how Wizard of Oz experiments involving multimodal systems differ from those involving voice-only systems. We then go on to discuss the Extended Wizard of Oz methodology and the Wizard of Oz testing environment and protocol that we have dev...

Multimodal interaction represents the future of human-computer interaction. It aims at enabling users to communicate more freely and naturally with automated systems. Current approaches to multimodal dialogue systems are based on two different strategies to perform the modality fusion. The former implements the multimodal fusion at dialogue management level, whereas the latter at grammar level....

In order to develop easy-to-use multimodal interfaces for mobile applications, effective usability evaluation methods (UEMs) are an essential component of the development process. Over the past decades various usability evaluation methods have been developed and implemented to improve and assure easyto-use user interfaces and systems. However, most of the so-called ‘classical’ methods exhibit s...

This paper is related to the development of an innovative multimodal biometric identification system. Unimodal biometric systems often face significant limitations due to sensitivity to noise intraclass variability and other factors. Multimodal biometric identification systems aim to fuse two or more physical or behavioral traits to provide optimal False Acceptance Rate (FAR) and False Rejectio...

We present the results of an investigation on software support for the SMUIML multimodal user interaction description language (UIDL). In particular, we introduce a graphical UIDL editor for the creation of SMUIML scripts. The data management as well as the dialogue modelling in our graphical editor is based on the SMUIML language. Due to the event-centered nature of SMUIML, the multimodal dial...

In this paper the project MobilTel is presented. The communication itself is becoming a multimodal interactive process. The MobilTel project provides research and development activities in multimodal interfaces area. The result is a functional architecture for mobile multimodal telecommunication system running on handheld device. The MobilTel communicator is a multimodal Slovak speech and graph...

This paper focuses on the estimation of reliability of unimodal and multimodal verification decisions produced by biometric systems. Reliability estimates have been demonstrated to be an elegant tool for incorporating quality measures into the process of estimating the probability of correctness of the decisions. In this paper we compare decisionand score-level schemes of multimodal fusion usin...

In this paper we present an extension of the EPFL’s Rapid Dialogue Prototyping Methodology to include multimodality, and show how it can be applied in the design of a multimodal application, the Archivus system. We begin with an overview of the standard speechonly rapid dialogue prototyping methodology, followed by a discussion of the extensions implemented to include multimodal capabilities. W...

Our aim is to use the logic of public announcements and more generally dynamic epistemic logics as a logic of speech acts. To that end we start from a simple multimodal logic of beliefs and goals (without common belief), and add public announcements. We suppose that announcements do not modify goals. We then consider several variants of speech acts of assertive and directive force and provide a...