Adjusting for Multiple Testing in Decision Tree Pruning

Over tting is a widely observed pathology of induction algorithms. Over tted models contain unnecessary structure that re ects nothing more than chance variations in the particular data sample used to construct the model. Portions of these models are literally wrong, and can mislead users. Over tted models require more storage space and take longer to execute than their correctlysized counterparts. Finally, over tting has been shown to reduce the accuracy of induced models on new data [13, 6]. For induction algorithms that build decision trees [1, 12, 14], is a common approach to correct over tting. Pruning techniques take an induced tree, examine individual subtrees, and remove those subtrees deemed to be unnecessary. While pruning techniques can di er in several ways, their primary di erences concern the criteria used to judge sub-trees. Many criteria have been proposed, including statistical signi cance tests [8], pessimistic error estimates [14], and minimum description length calculations [11]. Most common pruning techniques, however, do not account for one potentially important factor | multiple testing. Multiple testing occurs whenever an induction algorithm examines several candidate models and selects the one that best accords with the data. Any search process necessarily involves multiple testing, and most common induction algorithms involve implicit or explicit search through a space of candidate models. In the case of decision trees, search involves examining many possible subtrees and selecting the best one. Pruning techniques need to account for the number of subtrees examined, because such multiple testing a ects the apparent accuracy of models on training data [7]. This paper examines the importance of adjusting for multiple testing. Speci cally, it examines the e ectiveness of one particular pruning method | . Bonferroni pruning adjusts the results of a standard signi cance test to account for the number of subtrees examined at a particular node of a decision tree. Evidence that bonferroni pruning leads to better models supports the hypothesis that multiple testing is an important cause of over tting. The next section brie y surveys several relevant approaches to decision tree pruning. Section 3 presents the results of an experiment comparing bonferroni pruning to other pruning techniques. The nal section discusses the implications of the experiment and the content of the full paper.