Bias Corrected ROC Analysis for Visualization of Classifier Performance
نویسندگان
چکیده
The .632 error estimator is a bias correction of the bootstrap estimator which leads to an underestimation of the error when the apparent error is zero. As a consequence Efron and Tibshirani (1997) developed the .632+ bootstrap error as a modification that can handle this case. We demonstrate properties and behavior of this error estimation technique. Furthermore, we show how to apply the bootstrap method to estimate the classifiers sensitivity and specificity and demonstrate a bootstrap based ROC analysis of classification performance. An adaptation of the .632+ technique to calculate bias corrected sensitivities is straightforward and leads to .632+ bootstrap estimated ROC curves. We employ a simulation study to examine this method and its performance.
منابع مشابه
Analysis and Visualization of Classifier Performance: Comparison under Imprecise Class and Cost Distributions
Applications of inductive learning algorithms to realworld data mining problems have shown repeatedly that using accuracy to compare classifiers is not adequate because the underlying assumptions rarely hold. We present a method for the comparison of classifier performance that is robust to imprecise class distributions and misclassification costs. The ROC convex hull method combines techniques...
متن کاملA Novel Scalable Multi-class ROC for Effective Visualization and Computation
This paper introduces a new cost function for evaluating the multi-class classifier. The new cost function facilitates both a way to visualize the performance (expected cost) of the multi-class classifier and a summary of the misclassification costs. This function overcomes the limitations of ROC in not being able to represent the classifier performance graphically when there are more than two ...
متن کاملComparison of three-class classification performance metrics: a case study in breast cancer CAD [5749-64]
Receiver Operating Characteristic (ROC) analysis is a widely used method for analyzing the performance of two-class classifiers. Advantages of ROC analysis include the fact that it explicitly considers the tradeoffs in sensitivity and specificity, includes visualization methods, and has clearly interpretable summary metrics. Currently, there does not exist a widely accepted performance method s...
متن کاملAnalysis and Visualization of Classifier Performance with Nonuniform Class and Cost Distributions
Applications of machine learning have shown repeatedly that the standard assumptions of uniform class distribution and uniform misclassification costs rarely hold. Little is known about how to select classifiers when error costs and class distributions are not known precisely at training time, or when they can change. We present a method for analyzing and visualizing the performance of classifi...
متن کاملFault diagnosis in a distillation column using a support vector machine based classifier
Fault diagnosis has always been an essential aspect of control system design. This is necessary due to the growing demand for increased performance and safety of industrial systems is discussed. Support vector machine classifier is a new technique based on statistical learning theory and is designed to reduce structural bias. Support vector machine classification in many applications in v...
متن کامل