On overfitting and post-selection uncertainty assessments

Overfitting and Diversity in Classification Ensembles based on Feature Selection

This paper addresses Wrapper-like approaches to feature subset selection and the production of classifier ensembles based on members with different feature subsets. The paper starts with the observation that if an insufficient amount of data is used to guide the Wrapper search then the feature selection will overfit the data. If the objective of the feature selection exercise is to build a bett...

On Method Overfitting

Benchmark problems should be hard. True. Methods for solving problems should be useful for more than just “beating” a particular benchmark. Truer still, we believe. In this paper, we examine the worth of the approach consisting of concentration on a particular set of benchmark problems, an issue raised by a recent paper by Ian Gent. We find that such a methodology can easily lead to publication...

Uncertainty in Risk Assessments: Concepts and Principles

Transductive Learning via Model Selection; Can Overfitting be Exploited?

A novel transductive learning algorithm is proposed, which is based on the use of model selection. In its simplest form there are k possible labels, m labeled points and one unlabeled point. One model is built for each possible classification of the unlabeled point yM+1 = Li, i = 1, ..., k, using all m+1 points and m + 1 labels. Any standard model selection criterion can then be applied to sele...

Post - processing the hybrid method for addressing uncertainty in risk assessments

In this journal, a “hybrid method” was proposed for the joint propagation of probability distributions (expressing variability) and possibility distributions (i.e., fuzzy numbers, expressing imprecision or partial ignorance) in the computation of risk. In order to compare the results of the hybrid computation (a random fuzzy set) to a tolerance threshold (a tolerable level of risk), a post-proc...