Non-convex boosting with minimum margin guarantees
نویسنده
چکیده
Many classification algorithms achieve poor generalization accuracy on “noisy” data sets. We introduce a new non-convex boosting algorithm BrownBoost-δ, a noiseresistant booster, that is able to significantly increase accuracy on a set of noisy classification problems. Our algorithm consistently outperforms the original BrownBoost algorithm, AdaBoost, and LogitBoost on simulated and real data. These results hold even when early stopping of convex boosters is employed, suggesting that convex boosters may be underfitting, not overfitting. The increase in performance is correlated with the novel soft-margin maximizing parameter δ. Furthermore, we find that BrownBoost-δ is able to increase the margin at the decision boundary while sacraficing margin of easily classified examples. This tradeoff may provide a theoretical margin-based justification of empirical increased accuracy.
منابع مشابه
Robust Boosting via Convex Optimization: Theory and Applications
In this work we consider statistical learning problems. A learning machine aims to extract information from a set of training examples such that it is able to predict the associated label on unseen examples. We consider the case where the resulting classification or regression rule is a combination of simple rules – also called base hypotheses. The so-called boosting algorithms iteratively find...
متن کاملAdaBoost and Forward Stagewise Regression are First-Order Convex Optimization Methods
Boosting methods are highly popular and effective supervised learning methods which combine weak learners into a single accurate model with good statistical performance. In this paper, we analyze two well-known boosting methods, AdaBoost and Incremental Forward Stagewise Regression (FSε), by establishing their precise connections to the Mirror Descent algorithm, which is a first-order method in...
متن کاملMargins, Shrinkage, and Boosting
This manuscript shows that AdaBoost and its immediate variants can produce approximate maximum margin classifiers simply by scaling step size choices with a fixed small constant. In this way, when the unscaled step size is an optimal choice, these results provide guarantees for Friedman’s empirically successful “shrinkage” procedure for gradient boosting (Friedman, 2000). Guarantees are also pr...
متن کاملGeneralized Boosting Algorithms for Convex Optimization
Boosting is a popular way to derive powerful learners from simpler hypothesis classes. Following previous work (Mason et al., 1999; Friedman, 2000) on general boosting frameworks, we analyze gradient-based descent algorithms for boosting with respect to any convex objective and introduce a new measure of weak learner performance into this setting which generalizes existing work. We present the ...
متن کاملBoosting with Abstention
We present a new boosting algorithm for the key scenario of binary classificationwith abstention where the algorithm can abstain from predicting the label of a point,at the price of a fixed cost. At each round, our algorithm selects a pair of functions,a base predictor and a base abstention function. We define convex upper boundsfor the natural loss function associated to th...
متن کامل