In this chapter, we present the Multimodal Warehouse Project, which aimed at applying multimodal interaction to a warehouse picking process. We provide an overview of the warehouse picking procedure as well as the overall architecture of the multimodal picking application and technologies applied to design the application. Furthermore, we describe the execution of user tests of the picking appl...

With the wide use of digit, internet and multimedia, Multimodal foreign language teaching is becoming a brand-new teaching model. Recent years, there has been a great number of experts and scholars doing research on foreign language teaching using Multimodal theories. However, he application of Multimodal theories in English vocabulary teaching is very rare to see. Therefore, the author tries t...

This paper focuses on evaluation techniques for multimodal software. We suggest how multimodal educational systems can help us and give a proposal for how to evaluate these systems. Our goal is to come up with human-computer interaction (HCI) evaluation techniques for multimodal systems to support affective education and adapt usability evaluation taking into consideration the cognitive walkthr...

As a new style of effective information presentations and a new multimodal information content production on the World Wide Web (WWW), multimodal presentation using interactive life-like agents with verbal conversation capability appears to be very attractive and important. For this purpose, we have developed Multimodal Presentation Markup Language (MPML), which allows many users to write attra...

Multimodal interfaces are the emerging technology that offers expressive, transparent, efficient, robust, and mobile human-computer interaction. In this paper, we described the speech/gesture based multimodal interface systematically from the human factors point of view. To design more practical and efficient multimodal interface, human factors issues such as user modeling, usability studies, s...

Multimodal segmentation, like the physicians in practice, uses several images, representing different features of the object under investigation, to make segmentation decisions. In this paper, a multimodal twodimensional segmentation method is introduced. The principle of this method is the combination of multimodal region growing and multimodal edge detection. The combining condition is univer...

Multimodal interfaces provide more flexible and compelling interaction and can enable public information kiosks to support more complex tasks for a broader community of users. MATCHKiosk is a multimodal interactive city guide in which users are free to interact using speech, pen, touch or multimodal inputs and the system responds by generating multimodal presentations that synchronize synthetic...

In a multimodal human-machine conversation, user inputs are often abbreviated or imprecise. Simply fusing multimodal inputs together may not be sufficient to derive a complete understanding of the inputs. Aiming to handle a wide variety of multimodal inputs, we are building a context-based multimodal interpretation framework called MIND (Multimodal Interpreter for Natural Dialog). MIND is uniqu...

The paper proposes a method for tabular deduction for multimodal logical grammar, extending a method for linear logic presented in (Hepple, 1996; Hepple, 1998). Tabular deduction for multimodal logical grammar avoids redundant computation like chart parsing does for CFGs. The method discussed here enables one to process any kind of multimodal logical grammar fragment. An implementation of the m...