Carefully Appoximated Bayes Factors for Feature Selection in MaxEnt Models

Feature selection is essentially a model selection problem. If we take a frequentist maximum likelihood approach, we will, in the limit, select all features (unless, as is typical, we apply some sort of “early stopping” critereon). Additionally, basing the next feature to selected solely on standard measures such as likelihood gain, we fail to account for the variance of the estimate of this feature. In this note, I carefully derive an approximation to the Bayes factor in the feature/model selection problem for maximum entropy models. See [2] for an introduction to the use of maximum entropy models in the natural language processing domain. The advantages to using a Bayesian criterea for model selection are numerous, but the two strongest are that (a) it enables us to take into account uncertainty in likelihood when adding new features and (b) it allows us to decide when to stop adding features.