User Satisfaction Task: A Proposal for NTCIR-7

Good test collections, coupled with good evaluation metrics, are very useful for evaluating Information Access systems efficiently. But useful to whom? The in vitro (or Cranfield) evaluation paradigm has been criticised, mainly because of the absence of the user. On the other hand, user-in-the-loop evaluations are expensive, unrepeatable and often inconclusive. In light of this, we propose a new task for NTCIR that aims to directly measure the correlation between user satisfaction and evaluation metric values. To this end, we plan to reuse NTCIR-5 and NTCIR-6 Japanese monolingual newspaper test collections from the crosslingual task. Our final goal is to design new evaluation metrics that accurately approximate user satisfaction scores.