7. Conclusion and Future Work Novel Objective Function for Improved Phoneme Recognition Using Time-delay Neural Networks. 5. Simulation Results

recognize 10 isolated letters and used artificial markers on the lips. No visual feature extraction was integrated into their model. Also of interest are some psychological studies about human speechreading and their approach to describe the human performance. This measurements could also be applied to the performance analysis of automated speechreading systems. Dodd and Campbell [3], and Demorest and Bernstein [2] did some valuable work in this area. We have shown how a state-of-the-art speech recognition system can be improved by considering additional visual information for the recognition process. This is true for optimal recording conditions but even more for non-optimal recording conditions as they usually exist in real world applications. Experiments were performed on the connected letter recognition task, but similar results can be expected for continuous speech recognition as well. Work is in progress to integrate not only the time independent weight sharing but also position independent weight sharing for the visual TDNN, in order to locate and track the lips. We are also on the way to largely increase our database in order to achieve better recognition rates and to train speaker independently. Investigations of different approaches are still in progress in order to combine visual and acoustic features and to apply different prepro-cessing to the visual data. ACKNOWLEDGEMENTS We appreciate the help from the DEC on campus research center (CEC) for the initial data acquisition. classify /b/ and /p/ based only on visual information would lead to recognition rates not better than guessing, or the net perhaps would get sensitive for features which are uncorelated to the produced speech. This leads to the design of a smaller set of visual distinguishable units in speech, so called " visemes ". We investigate a new set of 42 visemes and a 1-ton mapping from the viseme set to the phoneme set. The mapping is necessary for the combined layer, in order to calculate the combined acoustic and visual hyphotheses for the DTW layer. For example the hypotheses for /b/ and /p/ are built out of the same viseme /b_or_p/ but the different phonemes /b/ and /p/ respectly. Our database consists of 114 and 350 letter sequences spelled by two male speakers. They consist of names and random sequences. The first data set was split into 75 training and 39 test sequences (speaker msm). The second data set was split into 200 training and 150 test sequences (speaker mcb). …