Analysis and Modeling of Non-Native Speech for Automatic Speech Recognition

The performance of automatic speech recognizers has been observed to be dramatically worse for speakers with non-native accents than for native speakers. This poses a problem for many speech recognition systems, which need to handle both native and non-native speech. The problem is further complicated by the large number of non-native accents, which makes modeling separate accents difficult, as well as the small amount of non-native speech that is often available for training. Previous work has attempted to address this issue by building accent-specific acoustic and pronunciation models or by adapting acoustic models to a particular non-native speaker. In this thesis, we examine the problem of non-native speech in a speaker-independent, large-vocabulary, spontaneous speech recognition system for American English, in which a large amount of native training data and a relatively small amount of non-native data are available. We investigate some of the major differences between native and non-native speech and attempt to modify the recognizer to better model the characteristics of nonnative data. This work is performed using the summit speech recognition system in the jupiter weather information domain. We first examine the modification of acoustic models for recognition of non-native speech. We show that interpolating native and non-native models reduces the word error rate on a non-native test set by 8.1% relative to a baseline recognizer using models trained on pooled native and non-native data (a reduction from 20.9% to 19.2%). In the area of lexical modeling, we describe a small study of native and non-native pronunciation using manual transcriptions and outline some of the main differences between them. We then attempt to model non-native word pronunciation patterns by applying phonetic substitutions, deletions, and insertions to the pronunciations in the lexicon. The probabilities of these phonetic confusions are estimated from non-native training data by aligning automatically-generated phonetic transcriptions with the baseline lexicon. Using this approach, we obtain a relative reduction of 10.0% in word error rate over the baseline recognizer on the non-native test set. Using both phonetic confusions and interpolated acoustic models, we further reduce the word error rate to 12.4% below baseline. Finally, we describe a study of language model differences between native and non-native speakers in the jupiter domain. We find that, within the resolution of our analysis, language model differences do not account for a significant part of the degradation in recognition performance between native and non-native test speakers. Thesis Supervisor: James R. Glass Title: Principal Research Scientist