Black-Box/Glass-Box Evaluation in Shiraz
نویسندگان
چکیده
The Shiraz project included an evaluation component: two ‘glass-box’ evaluations have been performed during the project as well as a black-box evaluation at the end of the project. The evaluations were based on the use of a bilingual tagged test corpus of 3000 sentences. Evaluation tools were developed in order to automate the evaluation process. The glass-box evaluations included the evaluation of components of the MT system, and in particular the Persian morphological analyzer, the dictionary and the parser. The evaluation of the translations themselves (black-box evaluations) were performed manually on a subset of the test corpus. This paper outlines the problems encountered in trying to use these evaluations for development and testing purposes as well as traditional ‘off-line’ evaluations.
منابع مشابه
Comparative Human and Automatic Evaluation of Glass-Box and Black-Box Approaches to Interactive Translation Prediction
Interactive translation prediction (ITP) is a modality of computer-aided translation that assists professional translators by offering context-based computer-generated continuation suggestions as they type. While most state-of-the-art ITP systems follow a glass-box approach, meaning that they are tightly coupled to an adapted machine translation system, a black-box approach which does not need ...
متن کاملCooperation between black box and glass box approaches for the evaluation of a question answering system
For the past three years, the question answering system QALC, currently developed in our team, has been taking part in the Question Answering (QA) track of evaluation campaigns TREC (Text REtrieval Conference). In the QA track, each system is evaluated according to a black box approach: as input, a set of questions, and as output, for each question, five answers ranked with regard to decreasing...
متن کاملFly with the EAGLES: evaluation of the "ACCeSS" spoken language dialogue system
This paper reports the experiences we had in evaluating the ACCeSS system using the EAGLES evaluation metrics both at the input/output (black box evaluation) and component levels (glass box evaluation). We deliver an example of a complete evaluation of a continuous speech/mixed initiative system using these standards. Furthermore, we discuss some useful extensions to them.
متن کاملEin kombinierter Black-Box- und Glass-Box-Test
Beim Testen kommt der Wahl der Testfälle eine entscheidende Bedeutung zu, denn mit der Festlegung der Testfälle wird über die Chancen zur Fehlerentdeckung entschieden. Viele Untersuchungen gehen der Frage nach, ob beim Black-Box-Test oder beim Glass-Box-Test effektivere Testfälle entstehen. Heute ist sich die Literatur weitgehend einig, dass die beiden Testverfahren keine Alternativen bilden, s...
متن کاملReuse and Challenges in Evaluating Language Generation Systems: Position Paper
Although there is an increasing shift towards evaluating Natural Language Generation (NLG) systems, there are still many NLG-specific open issues that hinder effective comparative and quantitative evaluation in this field. The paper starts off by describing a task-based, i.e., black-box evaluation of a hypertext NLG system. Then we examine the problem of glass-box, i.e., module specific, evalua...
متن کامل