In this paper we review the Receiver Operating Characteristic ROC curve and the test statistic in relation to the analysis of a confusion matrix We then show how these two methods are related and propose an extension to the ROC curve so that it shows contours of values These contours can be used to provide further insight into the appropriate setting of the decision threshold for a particular a...

In receiver operating characteristic (ROC) analysis, the sampling variability can result in uncertainties of performance measures. Thus, while evaluating and comparing the performances of algorithms, the measurement uncertainties must be taken into account. The key issue is how to calculate the uncertainties of performance measures in ROC analysis. Our ultimate goal is to perform the significan...

Evaluating classifier performance with ROC curves is popular in the machine learning community. To date, the only method to assess confidence of ROC curves is to construct ROC bands. In the case of severe class imbalance, ROC bands become unreliable. We propose a generic framework for classifier evaluation to identify the confident segment of an ROC curve. Confidence is measured by Tango’s 95%-...

It has been known for over 40 years that there are two fundamentally different kinds of detection tasks in the theory of signal detectability. The Type 1 task is to distinguish between events defined independently of the observer; the Type 2 task is to distinguish between one's own correct and incorrect decisions about those Type 1 events. For the Type 1 task, the behavior of the detector can b...

Receiver operating characteristic (ROC) curves are widely used to measure the discriminating power of medical tests and other classification procedures. In many practical applications, the performance of these procedures can depend on covariates such as age, naturally leading to a collection of curves associated with different covariate levels. This paper develops a Bayesian heteroscedastic sem...

Dual-process models of the word-frequency mirror effect posit that low-frequency words are recollected more often than high-frequency words, producing the hit rate differences in the word-frequency effect, whereas high-frequency words are more familiar, producing the false-alarm-rate differences. In this pair of experiments, the authors demonstrate that the analysis of receiver operating charac...