This study proposes the use of avatars to generate nonverbal communication in a multimodal interactive system, mainly based on the affective or emotional aspects of this type of communication. The objective of this study is the use of avatars as an element for generating non-verbal communication, without taking a specific application in account, but rather analysing all their interaction possib...

Multimodal communication involves the co-occurrence of different communicative channels, including speech, eye gaze and facial expressions. The questions addressed in this study are how these modalities correlate and how they are aligned to the discourse structure. The study focuses on a map task scenario whereby participants coordinate a route on a map, while their speech, eye gaze, face and t...

In this paper we discuss the design of human-robot interaction focussing especially on social robot communication and multimodal information presentation. As a starting point we use the WikiTalk application, an open-domain conversational system which has been previously developed using a robotics simulator. We describe how it can be implemented on the Nao robot platform, enabling Nao to make in...

There is a great potential for combining speech and gestures to improve human computer interaction because this kind of communication resembles more and more the natural communication humans use every day with each other. Therefore, this paper is about the multimodal interaction consisting of speech and gestures in an intelligent room. The advantages of using multimodal systems are explained an...

This paper presents the basic rationale behind the development and testing of a multimodal communication aid especially designed for people suffering from global aphasia, and thus having severe expressive difficulties. The principle of the aid is to trigger patient associations by presenting various multimodal representations of communicative expressions. The aid can in this way be seen as a co...

Emotions are part of our lives. Emotions can enhance the meaning of our communication. However, communication with computers is still done by keyboard and mouse. In this humancomputer interaction there is no room for emotions, whereas if we would communicate with machines the way we do in face-to-face communication much information can be extracted from the context and emotion of the speaker. W...

The following paper discusses the challenges faced in designing a flexible, easy-to-use, cross-platform software tool for modeling, analyzing and retrieving meaning from multimedia data. In a project currently being undertaken at the Multimodal Analysis Lab, Interactive and Digital Media Institute (IDMI), at the National University of Singapore (see http://multimodal-analysis-lab.org/), social ...

In this paper, we introduce an architecture for multimodal communication between humans and computers engaged in a shared task. We describe a representative dialogue between an artificial agent and a human that will be demonstrated live during the presentation. This assumes a multimodal environment and semantics for facilitating communication and interaction with a computational agent. To this ...

Data-driven knowledge discovery is becoming a new trend in various scientific fields. In light of this, the goal of the present paper is to introduce a novel framework to study one interesting topic in cognitive and behavioral studies -multimodal communication between human-human and human-robot interaction. We present an overall solution from data capture, through data coding and validation, t...

Program in Experimental Psychology University of California Santa Cruz, CA 95064 ABSTRACT It has long been a hope, expectation, and prediction that speech would be the primary medium of communication between humans and machines. To date, this dream has not been realized. We predict that exploiting the multimodal nature of spoken language will facilitate the use of this medium. We begin our pape...