We address the problem of applying machine-learning classifiers in domains where incorrect classifications have severe consequences. In these domains we propose to apply classifiers only when their performance can be defined by the domain expert prior to classification. The classifiers so obtained are called reliable classifiers. In the article we present three main contributions. First, we est...

A synthesis of the empirical Bayes method and the method of estimating equations is used to combine individual receiver operating characteristic (ROC) area estimates from different studies of the same diagnostic test into a single estimate. This single estimate represents the population mean from which individual areas under the ROC curves were sampled. The only data needed to carry out the met...

In this paper we establish the empirical likelihood method for the two-sample case in a general framework. We show that the result of Qin and Zhao (2000) basically covers the following two-sample models: the differences of two sample means, smooth Huber estimators, distribution and quantile functions, ROC curves, probability-probability (P-P) and quantile-quantile (Q-Q) plots. Finally, the stru...

Results The patients with active disease had greater IL-6 levels than did the patients with inactive disease (p=0.002) and controls (p=0.006).The cutoff value for active disease obtained from the ROC curve was 8,33pg/ml. During the active stage of the disease IL-6 levels were significantly higher in both systemic and polyarticular patients than in the oligoarticular patients. Serum IL-6 concent...

This paper shows that when a classifier is evaluated with nonrandom test data, ROC curves differ from the ROC curves that would be obtained with a random sample. To address this bias, this paper introduces a procedure for plotting ROC curves that are inferred from nonrandom test data. I provide simulations and an example with wine data to illustrate the procedure as well as the magnitude of bia...

BACKGROUND Ultrasonography (US) is a useful tool for breast imaging, yet is highly operator-dependent. OBJECTIVES We evaluated inter-observer variability and performance discrepancies between faculty members and radiology residents when describing breast lesions, by the fifth edition of breast imaging reporting and data system (BI-RADS)-US lexicon, and then attempted to identify whether inter...

OBJECTIVE To validate the SICK scoring system's ability to differentiate between individuals with higher and lower probabilities of death METHOD We performed a one year two-centre prospective evaluation of all children aged between one month and 12 years referred to the Paediatric team at St Stephens Hospital in Delhi and admitted to the Paediatric Department at West Middlesex University Hosp...

The ROC curve is one of the most common statistical tools useful to assess classifier performance. The selection of the best classifier when ROC curves intersect is quite challenging. A novel approach for model comparisons when ROC curves show intersections is proposed. In particular, the relationship between ROC orderings and stochastic dominance is investigated in a theoretical framework and ...

The area under the ROC curve (AUC) is a well-known measure of ranking performance, and is also often used as a measure of classification performance, aggregating over decision thresholds as well as class and cost skews. However, David Hand has recently argued that AUC is fundamentally incoherent as a measure of aggregated classifier performance and proposed an alternative measure [5]. Specifica...