Ensembles for Supervised Classification

This dissertation studies the use of multiple classi ers (ensembles or committees) in learning tasks. Both theoretical and practical aspects of combining classi ers are studied. We consider two di erent goals: The rst is to achieve better classi cation rates. We analyze both the representation ability of ensembles and algorithms that search for a solution in this representation space. Second, we consider the cost and time it takes to train classi ers. This leads us to consider systems that either learn with fewer labeled examples or learn while performing at their current ability. First we analyze the representational ability of voting ensembles. A voting ensemble may perform either better or worse than each of its individual members. We give tight upper and lower bounds on the classi cation performance of a voting ensemble as a function of the classi cation performances of its individual members. Boosting is a method of combining multiple \weak" classi ers to form a \strong" classi er. Several issues concerning boosting are studied in this thesis. We study SBA, a hierarchical boosting algorithm proposed by Schapire, in terms of its representation and its search. We show that if the weak learner has low representational complexity, SBA's search may fail to boost or may give a sub-optimal solution. We present a rejection boosting algorithm that trades-o exploration and exploitation: It requires fewer pattern labels at the expense of lower boosting ability. Ensembles may be useful in gaining information. We study their use to minimize labeling costs of data and to enable improvements on performance over time. For that purpose a model for on-site learning is presented. The system learns by querying \hard" patterns while classifying \easy" ones. This model is related to query-based ltering methods, but takes into account that, in addition to labeling, ltering through the data has a cost. The Query-By-Committee algorithm is used as a good approximator of the model space