In this paper we have proposed an automatic speech recognition framework using agents. In this we have included both audio recognition and visual recognition. The audio and visual modalities are complementary to each other and the combination of the two can improve the accuracy in affective user models. The audio features extracted are processed by audition agent. The visual processing agent ta...

Social affective expression is a main part of face-to-face interaction and it is highly linked to the language through the culture. This paper presents a study on Audio-Visual prosodic attitudes in Vietnamese, an under-resourced tonal language. Based on an audio-visual corpus of 16 attitudes, perception experiments were carried out with Vietnamese and French participants. The result analysis sh...

This paper presents results of training of coarticulation models for Czech audio-visual speech synthesis. Two approaches for solution of coarticulation in audio-visual speech synthesis were used, coarticulation based on dominance functions and visual unit selection. For both approaches, coarticulation models were trained. Models for unit selection approach were trained by visualy clustered data...

We have developed the Hand Motion Understanding (HMU) system that understands static and dynamic signs of the Australian Sign Language (Auslan). The HMU system uses a visual 3D hand tracker for motion sensing, and an adaptive fuzzy expert system for classification of the signs. This paper presents the hand tracker that extracts 3D hand configuration data with 21 degrees-of-freedom (DOFs) from a...

To represent an approach in video and audio configuration. Two modalities were used i.e., visual and audio tracks. From audio track, rhythms of music were described by segmentation and detection of music beats. From visual track, dancer’s features and trajectories were extracted to estimate rhythm of motion. Then configuration of visual and audio extractions were performed with two more applica...

This paper presents the design of an audio-visual speech corpus for in-vehicle audio-visual speech recognition. Throughout the world, there exist several audio-visual speech corpora. There are also several (audio-only) speech corpora for in-vehicle recognition. So far, we have not found an audiovisual speech corpus for in-vehicle speech recognition. And, we have not found any audio-visual speec...

The concept of using visual information as part of audio speech processing has been of significant recent interest. This paper presents a data driven approach that considers estimating audio speech acoustics using only temporal visual information without considering linguistic features such as phonemes and visemes. Audio (log filterbank) and visual (2D-DCT) features are extracted, and various c...

Automatic video scene change detection is a challenging task. Using audio or visual information alone often cannot provide a satisfactory solution. However, how to combine audio and visual information efficiently still remains a difficult issue since there are various cases in their relationship due to the versatility of videos. In this paper, we present an effective scene change detection meth...