Numerous studies have investigated the effectiveness of audio-visual multimodal learning for speech enhancement (AVSE) tasks, seeking a solution that uses visual data as auxiliary and complementary input to reduce noise noisy signals. Recently, we proposed lite (LAVSE) algorithm car-driving scenario. Compared conventional AVSE systems, LAVSE requires less online computation some extent solves u...

Abstract Audio-visual learning, aimed at exploiting the relationship between audio and visual modalities, has drawn considerable attention since deep learning started to be used successfully. Researchers tend leverage these two modalities improve performance of previously considered single-modality tasks or address new challenging problems. In this paper, we provide a comprehensive survey recen...

In this paper, we address the problem of automatic discovery of speech patterns using audio-visual information fusion. Unlike those previous studies based on single audio modality, our work not only uses the acoustic information, but also takes into account the visual features extracted from the mouth region. To improve the effectiveness of the use of multimodal information, several audio-visua...

In this paper, we investigate audio-visual interaction in sparse representation to obtain robust features for audio-visual speech recognition. Firstly, we introduce our system which uses sparse representation method for noise robust audio-visual speech recognition. Then, we introduce the dictionary matrix used in this paper, and consider the construction of audio-visual dictionary. Finally, we ...

Research shows that closing the eyes during retrieval can help both adults and children to remember more about witnessed events. In this study, we investigated whether the eye-closure effect in children is explained by general cognitive load, modality-specific interference, or a combination. 120 children (60 female) aged between 8 and 11 years viewed a 5-min clip depicting a theft and were ques...

This commentary is a review of the findings and ideas reported in the preceding nine articles on the effects of distraction on aspects of cognitive performance. The articles themselves deal with the disruptive effects of distraction on recall of words, objects and events, also on visual processing, category formation and other cognitive tasks. The commentary assesses the part played by "domain-...

Recognition of expressed emotion from speech and facial gestures was investigated in experiments on an audio-visual emotional database. A total of 106 audio and 240 visual features were extracted and then features were selected with Plus l-Take Away r algorithm based on Bhattacharyya distance criterion. In the second step, linear transformation methods, principal component analysis (PCA) and li...