On the Generalisation of Soft Margin Algorithms
نویسندگان
چکیده
Generalisation bounds depending on the margin of a classiier are a relatively recent development. They provide an explanation of the performance of state-of-the-art learning systems such as Support Vector Machines (SVM) and Adaboost. The diiculty with these bounds has been either their dependence on the minimal margin or their agnostic form. The paper presents a technique for correcting those points that fail to meet a target margin, hence creating a large margin classiier at the expense of additional functional complexity. Analysis of this technique leads to bounds that motivate the previously heuristic soft margin SVM algorithms as well as justifying the use of the quadratic loss in neural network training algorithms. The results are extended to give bounds for the probability of failing to achieve a target accuracy in regression prediction.
منابع مشابه
Bayes Point Machines
Kernel-classifiers comprise a powerful class of non-linear decision functions for binary classification. The support vector machine is an example of a learning algorithm for kernel classifiers that singles out the consistent classifier with the largest margin, i.e. minimal real-valued output on the training sample, within the set of consistent hypotheses, the so-called version space. We suggest...
متن کاملBoosting strategy for classification
This paper introduces a strategy for training ensemble classifiers by analysing boosting within margin theory. We present a bound on the generalisation error of ensembled classifiers in terms of the 2-norm of the margin slack vector. We develop an effective, adaptive and robust boosting algorithm, DMBoost, by optimising this bound. The soft margin based quadratic loss function is insensitive to...
متن کاملMaximum Margin Decision Surfaces for Increased Generalisation in Evolutionary Decision Tree Learning
Decision tree learning is one of the most widely used and practical methods for inductive inference. We present a novel method that increases the generalisation of genetically-induced classification trees, which employ linear discriminants as the partitioning function at each internal node. Genetic Programming is employed to search the space of oblique decision trees. At the end of the evolutio...
متن کاملSimple Incremental One-Class Support Vector Classification
We introduce the OneClassMaxMinOver (OMMO) algorithm for the problem of one-class support vector classification. The algorithm is extremely simple and therefore a convenient choice for practitioners. We prove that in the hard-margin case the algorithm converges with O(1/ √ t) to the maximum margin solution of the support vector approach for one-class classification introduced by Schölkopf et al...
متن کاملReliability of Cross-Validation for SVMs in High-Dimensional, Low Sample Size Scenarios
A Support-Vector-Machine (SVM) learns for given 2-classdata a classifier that tries to achieve good generalisation by maximising the minimal margin between the two classes. The performance can be evaluated using cross-validation testing strategies. But in case of low sample size data, high dimensionality might lead to strong side-effects that can significantly bias the estimated performance of ...
متن کامل