Following the ecological approach to visual perception, this paper investigates multimodal referring acts in HumanComputer Interaction. Preliminary results from a simulation experiment allow to: (a) clarify the effect perceptual organization on multimodal communication; (b) provide guidelines for designing effective multimodal interfaces. Demonstrating that both the verbal and the gestural part...

background many multimodal analgesia techniques have been tried to provide adequate analgesia for midline incisions extending above and below the umbilicus aiming at limiting the perioperative use of morphine thus limiting side effects. ultrasound (us) guidance made the anesthesiologist reconsider old techniques for wider clinical use. the rectus sheath block (rsb) is a useful technique under-u...

One of the most important motivations for many humanoid robot projects is that robots with a human-like body and human-like senses could in principle be capable of intuitive multimodal communication with people. The general idea is that by mimicking the way humans interact with each other, it will be possible to transfer the efficient and robust communication strategies that humans use in their...

The purpose of this study was to examine the delay effects in audiovisual speech perception for natural and synthetic faces. The main focus was on the SYNFACE project, the development of a telephone communication aid for hearing impaired persons. In the experiments, the consequence of temporal displacement of the audio in relation to the visual channel was investigated. The audio channel was na...

In recent years, with the spread of the household robots, the necessity to enhance the communication capabilities of those robot to people has been increasing. The objective of this study is to build a framework for a dialogue system dealing with multimodal information that a robot observes. We have applied partially observable Markov Decision Process to modeling multimodal interaction between ...

This chapter aims at describing how speech is multimodal not only in its perception but also in its production. It first focuses on multimodal perception of speech segments and speech prosody. It describes how multimodal perception is linked to speech production and explains why we consider speech perception as a sensory-motor process. It then analyses some aspects of hand-mouth coordination in...

The Catchment Feature Model (CFM) addresses two questions in multimodal interaction: how do we bridge video and audio processing with the realities of human multimodal communication, and how information from the different modes may be fused. We discuss the need for our model, motivate the CFM from psycholinguistic research, and present the Model. In contrast to ‘whole gesture’ recognition, the ...

A variety of theoretical frameworks predict the resemblance of behaviors between two people engaged in communication, in the form of coordination, mimicry, or alignment. However, little is known about the time course of the behavior matching, even though there is evidence that dyads synchronize oscillatory motions (e.g., postural sway). This study examined the temporal structure of nonoscillato...

Orangutans select different tactics for repairing failed communication, depending upon how well they are understood: they repeat signals if they are partially understood and switch tactics entirely if completely misunderstood.