Multimodal interfaces enable more natural and effective humancomputer interaction by providing multiple channels through which input or output may pass. In order to realize their full potential, they need to support not just input from multiple modes, but synchronized integration of semantic content from different modes. This paper describes a multimodal language processing architecture which a...

The design of applications with multimodal interfaces currently implies complex handcrafting by interface experts, lack of compliance with industry standards of cost effectiveness, maintenance and user focus such as those achieved by the current User-Centered Design methods. This paper presents an initial step towards a design by-example approach, whereby the end-user’s multimodal preferences f...

The development of interfaces has been a technology-driven process. However, the newly developed multimodal interfaces are using recognition-based technologies that must interpret human-speech, gesture, gaze, movement patterns, and other behavioral cues. As a result, the interface design requires a human-centered approach. In this paper we review the major approaches to multimodal Human Compute...

This paper deals with a multimodal annotation scheme dedicated to the study of gestures in interpersonal communication, with particular regard to the role played by multimodal expressions for feedback, turn management and sequencing. The scheme has been developed under the framework of the MUMIN network and tested on the analysis of multimodal behaviour in short video clips in Swedish, Finnish ...

In this paper we study how to learn stochastic, multimodal transition dynamics in reinforcement learning (RL) tasks. We focus on evaluating transition function estimation, while we defer planning over this model to future work. Stochasticity is a fundamental property of many task environments. However, discriminative function approximators have difficulty estimating multimodal stochasticity. In...

One of the major gaps in the current HTML5 web platform is the lack of interoperable means for an application to discover services and applications available in a given space and network. This problem is shared by the multimodal applications developed with web technologies, for example, in smart houses or applications for the Internet of Things. To address this gap, we produced a SOA approach f...

This paper presents a knowledge-based method of early-stage high level multimodal fusion of data obtained from speech input and visual scene. The ultimate goal is to develop a human-computer multimodal interface to assist elderly people living alone at home to perform their daily activities, and to support their active ageing and social cohesion. Crucial for multimodal high level fusion and suc...

Biometrics refers to automatic recognition of individuals based on their physiological and/or behavioral characteristics. Unimodal biometric systems allow person recognition based on a single source of biometric information but cannot guaranty a perfect identification. Such systems are sensitive to noisy sensor data, non-universality and lack of individuality of the chosen biometric trait, and ...

In this work we show that there is a need of using multimodal resources during human-computer interaction (HCI) in intelligent systems. We propose that not only creating multimodal output for the user is important, but to take multimodal input resources into account for the decision when and how to interact. Especially the use of multimodal input resources for the decision when and how to provi...

Biometrics is a science of recognizing a person based on his / her physiological or behavioral characteristics. So it plays an important role in identification and authenticating a person for immigration, physical, Forensic and Computer society. We can identify a person by using his / her any one or more than one of the characteristic traits and extract the feature set from the acquired data, a...