Multi-Angle Lipreading with Angle Classification-Based Feature Extraction and Its Application to Audio-Visual Speech Recognition

Recently, automatic speech recognition (ASR) and visual (VSR) have been widely researched owing to the development in deep learning. Most VSR research works focus only on frontal face images. However, assuming real scenes, it is obvious that a system should correctly recognize spoken contents from not but also diagonal or profile faces. In this paper, we propose novel method applicable faces taken at any angle. Firstly, view classification carried out estimate angles. Based results, feature extraction then conducted using best combination of pre-trained models. Next, lipreading features. We developed audio-visual (AVSR) addition conventional ASR. Audio results were obtained ASR, followed by incorporating audio decision fusion manner. evaluated our methods OuluVS2, multi-angle database. confirmed approach achieved performance among schemes phrase task. addition, found AVSR are better than ASR results.