Multimodal dialogue systems exploit one of the major characteristics of humanhuman interaction: the coordinated use of different modalities. Allowing all of the modalities to refer to and depend upon each other is a key to the richness of multimodal communication. We introduce the notion of symmetric multimodality for dialogue systems in which all input modes (e.g., speech, gesture, facial expr...

Human-computer interface designers are able to select from a rapidly increasing number of input/output modalities for a given application. Intuitively, the selection process can be viewed as the mapping of information from the task domain of the artefact to be built onto a set of input/output modalities for this artefact. Inherent to the mapping process is the goal of optimising the representat...

Automatically captioning images with natural language sentences is an important research topic. State of the art models are able to produce human-like sentences. These models typically describe the depicted scene as a whole and do not target specific objects of interest or emotional relationships between these objects in the image. However, marketing companies require to describe these importan...

Document layout analysis is crucial for understanding document structures. On this task, vision and semantics of documents, relations between components contribute to the process. Though many works have been proposed exploit above information, they show unsatisfactory results. NLP-based methods model as a sequence labeling task insufficient capabilities in modeling. CV-based detection or segmen...

Flowers attract pollinating insects like honeybees by sophisticated compositions of olfactory and visual cues. Using honeybees as a model to study olfactory-visual integration at the neuronal level, we focused on mushroom body (MB) output neurons (MBON). From a neuronal circuit perspective, MBONs represent a prominent level of sensory-modality convergence in the insect brain. We established an ...