Reliable Classifications with Machine Learning

In the past decades Machine Learning algorithms have been successfully used in several classification problems. While they often significantly outperform domain experts (in terms of classification accuracy or otherwise), they are mostly not being used in practice. A plausible reason for this is that it is difficult to obtain an unbiased estimation of a single classification’s reliability. While most Machine Learning algorithms can provide a quantitative assessment of single classifications (e.g. rule coverage, leaf class distribution), these may be heavily biased and therefore unreliable due to sparse training data and model limitations). In this paper we propose a general transductive method for estimation of classification’s reliability that is independent of applied Machine Learning algorithm. We compare our method with existing approaches and discuss its advantages. We perform extensive testing on 14 domains and 6 Machine Learning algorithms and show that our approach can frequently yield more than 100% improvement in reliability estimation performance.