Approximations to the MMI criterion and their effect on lattice-based MMI

Although maximum mutual information (MMI) training has been used for hidden Markov model (HMM) parameter estimation for more than twenty years ([2], [8], [5], [9], and [14]), it has recently become an essential part of the acoustic modeling repertoire thanks to the refinements introduced by Woodland and Povey ([16] and [11]). The earliest incarnations of MMI worked well on small vocabulary tasks with small models, for example digit recognition. However, one can expect to gain 10-20% in recognition accuracy over standard maximum likelihood methods regardless of the size of the task or the models when using the current methodology, lattice-based MMI. The machinery of lattice-based MMI consists of a model selection criterion called the MMI criterion and an iterative estimation algorithm called the extended Baum-Welch algorithm. This machinery is analogous to – it is in fact based on – the standard machinery used for maximum likelihood estimation with HMMs, where the model selection criterion is the log-likelihood of the training data and the iterative estimation algorithm is the BaumWelch algorithm ([3]). In both cases the estimation algorithm operates on the space of all possible model parameters by producing a new estimate of model parameters from an original estimate. Also, both of these estimation algorithms have been designed so that the model selection criterion is larger on the new estimate than it was on the original estimate. Finally, in both cases the machinery is operated in the same manner: starting from a choice of initial model parameters, we repeatedly apply the estimation algorithm, first to the initial choice, next to the result of this, etc., thereby creating a