Machine Learning Predicting nearly as well as the best pruning of a decision tree

Many algorithms for inferring a decision tree from data involve a two phase process First a very large decision tree is grown which typically ends up over tting the data To reduce over tting in the second phase the tree is pruned using one of a number of available methods The nal tree is then output and used for classi cation on test data In this paper we suggest an alternative approach to the pruning phase Using a given unpruned decision tree we present a new method of making predictions on test data and we prove that our algorithm s performance will not be much worse in a precise technical sense than the predic tions made by the best reasonably small pruning of the given decision tree Thus our procedure is guaranteed to be competitive in terms of the quality of its predictions with any pruning al gorithm We prove that our procedure is very e cient and highly robust Our method can be viewed as a synthesis of two previously studied techniques First we apply Cesa Bianchi et al s results on predicting using expert advice where we view each pruning as an expert to obtain an algorithm that has provably low prediction loss but that is com putationally infeasible Next we generalize and apply a method developed by Buntine and Willems Shtarkov and Tjalkens to derive a very e cient implementation of this procedure