On Supervised Selection of Bayesian Networks

Given a set of possible models (e.g., Bayesian network structures) and a data sample, in the unsupervised model selection problem the task is to choose the most accurate model with respect to the domain joint probabil­ ity distribution. In contrast to this, in su­ pervised model selection it is a priori known that the chosen model will be used in the future for prediction tasks involving more "focused" predictive distributions. Although focused predictive distributions can be pro­ duced from the joint probability distribu­ tion by marginalization, in practice the best model in the unsupervised sense does not ne­ cessarily perform well in supervised domains. In particular, the standard marginal likeli­ hood score is a criterion for the unsupervised task, and, although frequently used for super­ vised model selection also, does not perform well in such tasks. In this paper we study the performance of the marginal likelihood score empirically in supervised Bayesian net­ work selection tasks by using a large num­ ber of publicly available classification data sets, and compare the results to those ob­ tained by alternative model selection criteria, including empirical crossvalidation methods, an approximation of a supervised marginal likelihood measure, and a supervised version of Dawid's prequential (predictive sequential) principle. The results demonstrate that the marginal likelihood score does not perform well for supervised model selection, while the best results are obtained by using Dawid's prequential approach.