Improved bimodal speech recognition using tied-mixture HMMs and 5000 word audio-visual synchronous database

This paper presents methods to improve speech recognition accuracy by incorporating automatic lip reading. The paper improves lip reading accu­ racy by following approaches; 1)collection of im­ age and speech synchronous data of 5240 words， 2)feature extraction of 2・dimensional power spect日 around a mouth and 3)sub-word unit HMMs with tied-mixture distribution(Tied-Mixture HMMs). Ex­ periments through 100 word test show the perfor­ mance of 85% by lipreading alone. It is also shown that tied-mixture HMMs improve the lip reading ac­ curacy. The speech recognition experiments are car­ ried out over various SNR integrating audiovisual information. The results show the integration always realizes better performance than that using either au­ dio or visual information.