Evaluation of recommender systems: A multi-criteria decision making approach

The evaluation and selection of recommender systems is a difficult decision making process. This difficulty is partially due to the large diversity of published evaluation criteria in addition to lack of standardized methods of evaluation. As such, a systematic methodology is needed that explicitly considers multiple, possibly conflicting metrics and assists decision makers to evaluate and find the best recommender system among a given set of alternatives. This paper introduces Multi-Criteria Decision Making (MCDM) approach for evaluation of recommender systems. In particular, this paper proposes the use of Data Envelopment Analysis (DEA) approach, as a sub-category of MCDM, in order to solve this problem. Various DEA models are introduced and their applicability are illustrated. A real case of evaluation of recommender systems is used to demonstrate the approach.