MRE and Heteroscedasticity: An Empirical Validation of the Assumption of Homoscedasticity of the Magnitude of Relative Error

The Magnitude of Relative Error, MRE, is probably the most widely used basic metric in common evaluation criteria, e.g. MMRE and PRED, to assess the performance of software prediction models. The implicit rationale of using a relative error measure, rather than an absolute one, is to have a measure that is independent of scale. In this paper, we investigate if this implicit claim holds true using a data set of 81 ERP projects. The results suggest that MRE is not independent of scale. Rather, MRE is larger for small projects than for large projects. A practical consequence is that a project manager predicting a small ERP project will falsely believe in a too high accuracy in terms of MMRE. Vice versa when predicting a large ERP project.