The growing emphasis on multimodal interface design is fundamentally inspired by the aim to support natural, flexible, efficient, and powerfully expressive means of human-computer interaction that are easy to learn and use. Multimodal interfaces represent a new direction for computing that draws from the myriad input and output technologies becoming available, and that potentially can integrate...

The deep superior colliculus (DSC) integrates multisensory input and triggers an orienting movement toward the source of stimulation (target). It would seem reasonable to suppose that input of an additional modality should always increase the amount of information received by a DSC neuron concerning a target. However, of all DSC neurons studied, only about one half in the cat and one-quarter in...

The objective of our work is the development of a natural language dialogue system for information retrieval with multimodal input and multimedia output. Overall, the system consists of three phases: input analysis, information and knowledge management and output generation. The dialogue system is designed for consulting old Mexican historical documents. In this paper we describe the designed a...

In the last decades, many efforts have been made to solve multimodal optimization problems using Particle Swarm Optimization (PSO). To produce good results, these PSO algorithms need to specify some niching parameters to define the local neighborhood. In this paper, our motivation is to propose the novel neighborhood structures that remove undesirable niching parameters without sacrificing perf...

In this paper, multimodal learning for facial expression recognition (FER) is proposed. The multimodal learning method makes the first attempt to learn the joint representation by considering the texture and landmark modality of facial images, which are complementary with each other. In order to learn the representation of each modality and the correlation and interaction between different moda...

This paper describes a new application framework (OpenMAF) for rapid development of multimodal applications in computer-aided medicine. MAF applications are multimodal in data, in representation, and in interaction. The framework supports almost any type of biomedical data, including DICOM datasets, motion-capture recordings, or data from computer simulations (e.g. finite element modeling). The...

Mobile broadband internet access and powerful mobile devices make interesting and novel communication applications possible (e.g., recently emerging VoIP applications). Additionally, speech recognition has matured to the point that companies can seriously consider its use. We developed a distributed framework that enables multimodal user interfaces with speech recognition (dictation and command...

This paper presents Agile Planner for Digital Tabletops (APDT) as a tool that facilitates agile planning meetings using large horizontal displays. Utilizing APDT on a reasonably sized digital tabletop allows collaborators to create, edit, move, rotate, toss and delete index cards just like they would do with paper artifacts. APDT provides a multimodal input system that supports gesture-, handwr...

We propose an architecture of neural network that can learn and integrate sequential multimodal information using Long Short Term Memory. Our model consists of encoder and decoder LSTMs and multimodal autoencoder. For integrating sequential multimodal information, firstly, the encoder LSTM encodes a sequential input to a fixed range feature vector for each modality. Secondly, the multimodal aut...