Scalable Recurrent Neural Network Language Models for Speech Recognition

Language Modelling is a crucial component in many areas and applications including automatic speech recognition (ASR). n-gram language models (LMs) have been the dominant technology during the last few decades, due to their easy implementation and good generalism on unseen data. However, there are two well known problems with n-gram LMs: data sparsity; and the n-order Markov assumption. Previous research has explored various options to mitigate these issues. Recently, recurrent neural network LMs (RNNLMs) have been found to offer a solution for both of these issues. The data sparsity issue is solved by projecting each word into a low, continuous, space, and the long term history is modelled via the recurrent connection between hidden and input layer. Hence, RNNLMs have become increasingly popular and promising results have been reported on a range of tasks. However, there are still several issues to be solved in area to apply RNNLMs to the ASR task. Due to the long term history, the training of RNNLMs is difficult to parallelise and slow to train on large quantities of training data and large model size. It is easy to apply Viterbi decoding or lattice rescoring for standard n-gram LMs as they have limited history, while it is difficult for RNNLMs because of their long term history. This thesis aims to facilitate the application of RNNLMs in ASR. First, efficient training and evaluation of RNNLMs are developed. By splicing multiple sentences, RNNLMs could be trained efficiently with bunch (i.e. minibatch) mode on GPUs. Several improved training criteria are also investigated to further improve the efficiency of training and evaluation. Second, two algorithms are proposed for efficient lattice rescoring and compact lattices are able to generate. Third, the adaptation of RNNLMs is investigated. Model fine-tune and incorporation of informative feature based adaptation are investigated. Various topic models are applied to extract topic representation for efficient adaptation. Finally, the different modelling power of RNNLMs and n-gram LMs are explored and the interpolation of these two types of models is studied. The first contribution of this thesis is the efficient training and inference of RNNLMs. The training of RNNLMs is computationally heavy due to the large output layer and difficulty of parallelisation. In most previous works, RNNLMs were trained on CPU with class based RNNLMs. In this thesis, a novel sentence splicing method is proposed, which allows RNNLMs to be trained much more efficiently with bunch mode. GPU is also used