The era in which we live in today is dominated by technology and digital interactions. Consequently, our desire for cutting-edge technology is spawning an increasing demand to continually build new systems. More recently this demand has geared towards building robot systems to be implemented in our everyday lives. Yet, before we can make this idea a reality we must first look at taking strides ...

Multimodal interfaces offer its users the possibility of interacting with computers, in a transparent, natural way, by means of various modalities. Fusion engines are key components in multimodal systems, responsible for combining information from different sources and extract a semantic meaning from them. This fusion process allows many modalities to be effectively used at once and therefore a...

This paper discusses innovative techniques to automatically estimate a user's emotional state analyzing the speech signal and haptical interaction on a touch-screen or via mouse. The knowledge of a user's emotion permits adaptive strategies striving for a more natural and robust interaction. We classify seven emotional states: surprise, joy, anger, fear, disgust, sadness, and neutral user state...

In this paper, we review recent research that examines audio-visual integration in multimodal communication. The topics include bimodality in human speech, human and automated lip-reading, facial animation, lip synchronization, joint audio-video coding, and bimodal speaker verification. We also study the enabling technologies for these research topics, including automatic facial feature trackin...

The overall behavior and nature of complex natural systems is in large part determined by the number and variety of the mechanisms involved – and the complexity of their interactions. Embodied natural communication belongs to this class of systems, encompassing many cognitive mechanisms that interact in highly complex ways, both within and between communicating individuals, constituting a heter...

We present a multimodal interaction framework that allows a human operator to interact with co-located drones during search and rescue missions. In contrast with usual human-multidrones interaction scenarios, in this case the operator is not fully dedicated to the control of the robots, but directly involved in search and rescue tasks, hence only able to provide fast, although high-value, instr...

Recently, there has been an increased interest in the generation of multimodal referring expressions (e.g., Salmon-Alt and Romary, 2000; Lester et al., 1999; André and Rist, 1996; Reithinger, 1992; Claassen, 1992). The task involved in generation of multimodal referring expressions is to decide what is the best way to refer to a target via different modalities in the current context. Existing a...

Spiders perceive the world using multiple sensory modes, including vibration, vision, and chemical senses, for prey detection and communication. These sensory modes are used in many communication contexts, either individually or in multimodal signaling. Selection for effective signaler-receiver communication and species discrimination is especially strong for these predatory and potentially can...

Virtual Environments (VE) represent advanced human computer interaction systems, communicating over several channels of information. The basic terms we need to describe human VE interaction are “communication channel” and “multimodality”. The communication channel is as a pathway between the user and the simulation that mediates the interaction. Multimodality is the quality of a system, which a...

The grand challenge of multimodal interface creation is to build reliable processing systems able to analyze and understand multiple communication means in real-time. This opens a number of associated issues covered by this chapter, such as heterogeneous data types fusion, architectures for real-time processing, dialog management, machine learning for multimodal interaction, modeling languages,...