Evaluating Software Project Prediction Systems

The problem of developing usable software project cost1 prediction systems is perennial and there are many competing approaches. Consequently, in recent years there have been exhortations to conduct empirically based evaluations in order that our understanding of project prediction might be based upon real world evidence. We now find ourselves in the interesting position of possessing this evidence in abundance. For example, a review of just three software engineering journals identified 50 separate studies [7] and overall several hundred studies have been published. This naturally leads to the next step of needing to construct a body of knowledge, particularly when not all evidence is consistent. This process of forming a body of knowledge is generally referred to as metaanalysis. It is an essential activity if we are to have any hope of making sense of, and utilising, results from our empirical studies. However, it becomes apparent that when systematically combining results many difficulties are encountered.