Automatic Selection of Reliability Estimates for Individual Regression Predictions Using Meta-Learning and Internal Cross-Validation

In machine learning and its risk-sensitive applications (e.g. medicine, engineering, business), the reliability estimates for individual predictions provide more information about the individual prediction error than the average accuracy of predictive model (e.g. relative mean squared error). Furthermore, they enable the users to distinguish between more and less reliable predictions. The empirical evaluations of the existing individual reliability estimates revealed that the successful estimates’ performance depends on the used regression model and on the particular problem domain. In the current paper, we focus on that problem as such and propose and empirically evaluate two approaches for automatic selection of the most appropriate estimate for a given domain and regression model: the meta-learning approach and the internal cross-validation approach. The testing results of both approaches demonstrated an advantage in performance of dynamically chosen reliability estimates to the performance of the individual reliability estimates. The best results were achieved using the internal cross-validation procedure, where 73% of testing domains significantly positively correlated with the prediction error. In addition, the preliminary testing of the proposed methodology on a medical domain demonstrated the potential for its usage in practice.