Multimodal Corpora : How Should Multimodal Corpora Deal with the Situation ?
نویسندگان
چکیده
The paper introduces the notion of multimodal cluster (MMC). MMC is a multicomponent spoken unit, which includes diads “meaning + gesture”, “meaning + phonetic phenomenon” (double MMC) or triad “meaning + gesture + phonetic phenomenon” (triple MMC). All components of the same MMC are synchronized in the speech, gestural and phonetic components conveying the same idea as the semantic component (naturally, with available means). To put it another way, MMC is a combination of speech phenomena of different modi (semantic, visual, sound), which are tightly connected in the spoken language, and roughly speaking mean the same, i.e. convey the same idea by their own means. The paper describes some examples of double and triple MMCs, which are specific for the Spoken Russian.
منابع مشابه
Towards Metadata Descriptions for Multimodal Corpora of Natural Communication Data
Metadata play an important role for successful corpus management and reusability of corpora. For linguistic resources there already exist a large amount of metadata descriptions and metadata schemes. However, not much work has been done to develop metadata for the particular structure of multimodal corpora, yet. In this paper we provide a review of existing metadata profiles for multimodal data...
متن کاملThe SmartWeb Corpora: Multimodal Access to the Web in Natural Environments
As a result from the German SmartWeb project three speech corpora, one of them multimodal, have been published by the Bavarian Archive for Speech Signals (BAS). They contain speech and video signals from human–machine interactions in real indoor and outdoor environments. The scenarios for these corpora are a typicial handheld PDA interaction (SHC), an interaction on a running motorcycle (SMC) a...
متن کاملThe REX corpora: A collection of multimodal corpora of referring expressions in collaborative problem solving dialogues
This paper describes a collection of multimodal corpora of referring expressions, the REX corpora. The corpora have two notable features, namely (1) they include time-aligned extra-linguistic information such as participant actions and eye-gaze on top of linguistic information, (2) dialogues were collected with various configurations in terms of the puzzle type, hinting and language. After desc...
متن کاملFrom Annotated Multimodal Corpora to Simulated Human-Like Behaviors
Multimodal corpora prove useful at different stages of the development process of embodied conversational agents. Insights into human-human communicative behaviors can be drawn from such corpora. Rules for planning and generating such behavior in agents can be derived from this information. And even the evaluation of human-agent interactions can rely on corpus data from human-human communicatio...
متن کاملImage-Language Multimodal Corpora: Needs, Lacunae and an AI Synergy for Annotation
The growing demand for intelligent multimedia systems has led to the development of various multimodal resources and corresponding annotation schemes and processing tools. In this paper, we argue that there is a striking lack of multimodal corpora capturing the association and interaction of visual and linguistic data. We relate this research lacuna to vision-language integration prototypes dev...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2012