Workshop on W3c's Multimodal Architecture and Interfaces Kinesthetic Input Modalities for the W3c Multimodal Architecture

Deutsche Telekom Laboratories and T-Systems recently developed various multimodal prototype applications and modules allowing kinesthetic input, i.e. devices that can be moved around in order to alter the applications’ state. We developed first prototypes were using proprietary technologies and principles, whereas latest demonstrators more and more follow the W3C’s Multimodal Architecture. This paper describes how we propose to integrate kinesthetic input into the multimodal framework as a new modality and discusses how this component fits into the W3C Multimodal Architecture. Introduction Mobile devices are getting more and more powerful while at the same time they are becoming smaller and smaller. Most of today’s mobile handsets are equipped with cameras and there are first mobile phones shipped with acceleration sensors. These developments enable new input paradigms and capabilities, which provide intuitive input modes. Deutsche Telekom Laboratories and T-Systems have integrated various motion sensors based e.g. on acceleration sensors or analysis of a camera picture into prototypical applications, which could potentially profit from input gestures, in which the whole device is moved around. The motion patterns (e.g. tilt up, tilt down, tilt left, tilt right) are derived from raw sensor or camera output using digital signal processing. The output patterns are mapped to corresponding events and may be used within the application control logic as user input events to manipulate the user interface. In our first experiments, tilt up and down were proven to be effective for menu and list navigation. The general interaction logic is shown in the picture below. It follows a “sphere” metaphor, in that the motion patterns can be thought of as laying in the directions indicated on a virtual sphere (“tilt up” means tilting the upper part of the device down along a sphere’s surface): This paper describes the technical integration of the motion detection sensor into the multimodal application and discusses the possibility to use it as a modality component within the W3C Multimodal Architecture. In analogy to the VoiceXML use case referred to in the W3C Working Draft on Multimodal Architecture and Interface [MMIARCH], we also introduce the cases of local and remote processing and discuss their advantages and disadvantages. Kinesthetic modality component The digital sensor (or camera) together with the digital signal processing can be seen as a modality component within the W3C Multimodal Architecture. It generates events corresponding to user motion