This paper presents the results of the WMT12 shared tasks, which included a translation task, a task for machine translation evaluation metrics, and a task for run-time estimation of machine translation quality. We conducted a large-scale manual evaluation of 103 machine translation systems submitted by 34 teams. We used the ranking of these systems to measure how strongly automatic metrics cor...

Software design evaluation plays an important role in software development process, to generate software product with high levels of productivity and efficiency. To achieve high level of qualitative software product, implementation of pattern can create a domain specific framework to provide consistency throughout a software solution. This paper proposed an evaluation pattern called Criteria Ba...

Document-level Machine Translation (MT) has been drawing more and more attention due to its potential of resolving sentencelevel ambiguities and inconsistencies with the benefit of wide-range context. However, the lack of simple yet effective evaluation metrics largely impedes the development of such document-level MT systems. This paper proposes to improve traditional MT evaluation metrics by ...

Automatic evaluation metrics are fast and cost-effective measurements of the quality of a Machine Translation (MT) system. However, as humans are the end-user of MT output, human judgement is the benchmark to assess the usefulness of automatic evaluation metrics. While most studies report the correlation between human evaluation and automatic evaluation at corpus level, our study examines their...

It is time to make changes to the current research evaluation system, which is built on the journal selection. In this study, we propose the idea of continuous, dynamic and comprehensive article-level-evaluation based on article-level-metrics. Different kinds of metrics are integrated into a comprehensive indicator, which could quantify both the academic and societal impact of the article. At d...

We present the MT-NCD and MT-mNCD machine translation evaluation metrics as submission to the machine translation evaluation shared task (MetricsMATR 2010). The metrics are based on normalized compression distance (NCD), a general information theoretic measure of string similarity, and evaluated against human judgments from the WMT08 shared task. The experiments show that 1) our metric improves...

In the present work we study semi-automatic evaluation techniques of machine translation (MT) systems. These techniques are based on a comparison of theMT system’s output to human translations of the same text. Various metrics were proposed in the recent years, ranging from metrics using only a unigram comparison to metrics that try to take advantage of additional syntactic or semantic informat...

In the present work we study semi-automatic evaluation techniques of machine translation (MT) systems. These techniques are based on a comparison of the MT system’s output to human translations of the same text. Various metrics were proposed in the recent years, ranging from metrics using only a unigram comparison to metrics that try to take advantage of additional syntactic or semantic informa...

In this work we present the fundamentals of the IQMT framework for MT evaluation. IQMT offers a common workbench on which existing evaluation metrics can be utilized. We suggest the IQ measure and test it on the Chinese-toEnglish data from the IWSLT 2004 Evaluation Campaign. We show how the correlation with human assessments at the system level improves substantially for most individual metrics...

Several evaluation metrics have been proposed for topic segmentation. Most of them rely on the paradigm that segmentation is mainly a task that detects boundaries, and thus are oriented on boundary detection evaluation. Nevertheless, this paradigm is not appropriate to get homogeneous chapters, which is one of the major applications of topic segmentation. For instance on Broadcast News, topic s...