Multimodal interfaces, combining the use of speech, graphics, gestures, and facial expressions in input and output, promise to provide new possibilities to deal with information in more effective and efficient ways, supporting for instance: the understanding of possibly imprecise, partial or ambiguous multimodal input; the generation of coordinated, cohesive, and coherent multimodal presentatio...

This paper discusses software architectures of multimodal systems. The recent availability of new input technologies brought a whole new type of systems, able to support communication with the user through multiple interaction channels. Multimodal systems that allow modalities to be combined seem to be the most promising in the field of multimodal interaction. The aim is to overcome limitations...

Multimodal sentiment analysis is an increasingly popular research area, which extends the conventional language-based definition of sentiment analysis to a multimodal setup where other relevant modalities accompany language. In this paper, we pose the problem of multimodal sentiment analysis as modeling intra-modality and inter-modality dynamics. We introduce a novel model, termed Tensor Fusion...

The development of a multimodal Human-Robot Interface (HRI) involving mixed-initiative and context-awareness is complex and laborious. The integration of individual modalities (e.g., gesture recognition or speech output) and the design of natural human-robot interaction are two different tasks that both require their own expertise. In this paper, we consider three different roles that participa...

Multimodal interaction allows users to specify commands using combinations of inputs from multiple different modalities. For example, in a local search application, a user might say “gas stations” while simultaneously tracing a route on a touchscreen display. In this demonstration, we describe the extension of our cloud-based speech recognition architecture to a Multimodal Semantic Interpretati...

This paper introduces a new technique called adaptive elitistpopulation search method for allowing unimodal function optimization methods to be extended to efficiently locate all optima of multimodal problems. The technique is based on the concept of adaptively adjusting the population size according to the individuals’ dissimilarity and the novel elitist genetic operators. Incorporation of the...

Two additional figures are provided to demonstrate the superior performance of multimodal transfer and compare it to the state-of-the-art network by Johnson et al.(Johnson Net). In Fig. B, we show multimodal transfer results for many different styles on a large variety of contents. In Fig. C, we evaluate multimodal transfer on larger images (1800×1352) and compare the results with those of John...

This paper deals with the MUMIN multimodal annotation scheme (Allwood et al 2004), which was developed for the study of gestures and facial displays in interpersonal communication, with particular regard to the role played by multimodal expressions for feedback, turn management and sequencing. The scheme has been applied to the analysis of multimodal behaviour in short video clips in Swedish, F...

The design and implementation of multimodal interfaces that support a variety of modes to enable natural interaction is still limited. We propose a multimodal interaction web framework that considers on the one hand the current W3C standardization activities, such as Start Chart XML and the Model-Based UI Working Group. But on the other hand it implements recent research results enabling the di...

Usability evaluation of multimodal systems is a complex issue. Multimodal systems provide multiple channels to communicate with the system. Thus, the single modalities as well as their combination have to be taken into account. This paper aims to investigate how ratings of single modalities relate to the ratings of their combination. Therefore a usability evaluation study was conducted testing ...