Precision and Recall Without Ground Truth

In this paper we present a way to use precision and recall measures in total absence of ground truth. 1 Precision and Recall 1.1 General Definitions and Notation Precision Pr and Recall Rc (and often associated F-measure or ROC curves) are standard metrics expressing the quality of Information Retrieval methods [8]. They are usually expressed with respect to a query q (or averaged over a series of queries) over a data set ∆ such that: Pr q = ∣∣P∆ q ∩Rq ∣∣ ∣∣R∆q ∣∣ (1) Rcq = ∣∣P∆ q ∩Rq ∣∣ ∣∣P∆ q ∣∣ (2) where P q is the set of all documents in ∆, relevant to query q, and where Rq is the set of documents actually retrieved by q. Although we can make a safe assumption by considering Rq known (i.e the query q can actually be executed, and returns a known, manageable set of results), the same assumption does not always hold for P q , as will be shown later. For ease of reading we will refer to respectively Pr, P Rc, and R, when there is no ambiguity on ∆ and q. Often both are combined in the Fβ measure, where Fβ = ( 1 + β ) PrRc β2Pr +Rc (3) ∗Bart Lamiroy was a visiting scientist at Lehigh University in 2010-2011. This work was conducted at the Computer Science and Engineering Departent at Lehigh University and was supported in part by a DARPA IPTO grant administered by Raytheon BBN Technologies. 1 in ria -0 06 17 31 4, v er si on 1 26 A ug 2 01 1 Author manuscript, published in "Ninth IAPR International Workshop on Graphics RECognition GREC 2011 (2011)"