In this paper, we propose an automatic evaluation procedure based on a metric which could provide summary evaluation without human assistance. Our system, which includes two metrics, for summary evaluation, is presented. The first metric is based on a known and powerful statistical test, the χ goodness-of-fit test, and has been used in several applications. The second metric is derived from thr...

This paper proposes real-time performance evaluation metrics for the object detection and tracking algorithms used in intelligent video surveillance systems. Since an intelligent video surveillance system should support real-time performance, as well as the spatial accuracy of the object detection and tracking, the performance evaluation tool should evaluate not only the spatial accuracy, but a...

In this paper we propose a unified framework for automatic evaluation of NLP applications using N-gram co-occurrence statistics. The automatic evaluation metrics proposed to date for Machine Translation and Automatic Summarization are particular instances from the family of metrics we propose. We show that different members of the same family of metrics explain best the variations obtained with...

The strict character of most of the existing Machine Translation (MT) evaluation metrics does not permit them to capture lexical variation in translation. However, a central issue in MT evaluation is the high correlation that the metrics should have with human judgments of translation quality. In order to achieve a higher correlation, the identification of sense correspondences between the comp...

This paper discusses methods behind tracker evaluation, the aim being to evaluate how well a tracker is able to determine the position of a target object. Few metrics exist for positional tracker evaluation; here the fundamental issues of trajectory comparison are addressed, and metrics are presented which allow the key features to be described. Often little evaluation on how precisely a target...

Many automatic evaluation metrics for machine translation (MT) rely on making comparisons to human translations, a resource that may not always be available. We present a method for developing sentence-level MT evaluation metrics that do not directly rely on human reference translations. Our metrics are developed using regression learning and are based on a set of weaker indicators of fluency a...

Current methods for automatically evaluating grammatical error correction (GEC) systems rely on gold-standard references. However, these methods suffer from penalizing grammatical edits that are correct but not in the gold standard. We show that reference-less grammaticality metrics correlate very strongly with human judgments and are competitive with the leading reference-based evaluation metr...

We introduce a manually-created, multireference dataset for abstractive sentence and short paragraph compression. First, we examine the impact of singleand multi-sentence level editing operations on human compression quality as found in this corpus. We observe that substitution and rephrasing operations are more meaning preserving than other operations, and that compressing in context improves ...