Modality Theory is a comparatively new field of investigation which addresses the following general problem of mapping task domain information into interactive multimodal interfaces: given any particular set of information which needs to be exchanged between user and system during task performance in context, identify the input/ output modalities which constitute an optimal solution to the repr...

Modality theory addresses the following general problem of mapping task domain information into interactive multimodal interfaces: given any particular set of information which needs to be exchanged between user and system during task performance in context, identify the input/output modalities which constitute an optimal solution to the representation and exchange of that information. This pap...

Modality-specific naming deficits, such as optic aphasia, have been taken as evidence that semantics is organized into distinct modality-specific subsystems. We adopt an alternative view in which semantics is a learned, internal representation within a parallel distributed processing system that maps between multiple input and output modalities. We show that the robustness of a task to damage d...

Advances in information technologies and telecommunications are making available a very large number of possible input/output modality combinations which are potentially useful for the expression and exchange of information in human-human-system interaction (HHSI). However, standard or shared foundations consisting of all available and possible elementary, or unimodal, input and output modaliti...

This paper advances a unified knowledge-based approach to the process of choosing the most appropriate modality or combination of modalities in multimodal output generation. We propose a Modality Ontology (MO) that models the knowledge needed to support the two most fundamental processes determining modality choice – modality allocation (choosing the modality or set of modalities that can best ...

Mobile multimodal dialogue systems allow the user and the system to adapt their choice of input and output modality according to various technical and cognitive resource limitations and the task at hand. We present the multimodal dialogue system SmartKom, that can be used as mobile travel companion for car drivers and pedestrians. SmartKom combines speech, gestures, and facial expressions for i...

In this paper a conceptual model for the design and construction of interactive multimodal systems is presented. This model is based on a representational language for the specification of dialogue models and its associated program interpreter. Dialogues models are domain and modality independent conversational schemes characterizing the dynamic of a multimodal interaction, and domain specific ...

Sonography has become an integral part of the care of renal allograft recipients. It is a simple, inexpensive and readily available non-invasive imaging modality. It is indicated as the initial investigation in patients presenting with decreased urine output, pain, infection and hematuria and for doing a percutaneous allograft biopsy. While sonography confirms the diagnosis of obstructive nephr...

This paper addresses usability challenges of speech and multimodal interfaces by investigating modality preferences and interaction quality of a mobile medication in-take reminder. Older users (normal hearing, hearing impairment) interacted with input and output modality conditions on a smartphone in a laboratory study. Interestingly hearing impaired users prioritized voice only interaction ove...