Traditional reliability metrics are based on probability measures. However, in engineering practices, failure data are often so scarce that traditional metrics cannot be obtained. Furthermore, in many applications, premises of applying these metrics are violated frequently. Thus, this paper will give some new reliability metrics which can evaluate products’ reliability with few failure data. Fi...

Measuring the perceived quality of printed images are important to assess the performance of printers and to evaluate technology advancements. Image quality metrics have been proposed to objectively assess the quality of images, and new metrics are continuously proposed. However, applying these metrics to printed images are not straightforward, since they require a digital input. We present a n...

BACKGROUND Medical Image segmentation is an important image processing step. Comparing images to evaluate the quality of segmentation is an essential part of measuring progress in this research area. Some of the challenges in evaluating medical segmentation are: metric selection, the use in the literature of multiple definitions for certain metrics, inefficiency of the metric calculation implem...

Results of subjective evaluations of perceived sound quality preferences for mobile telephone handsets with different acoustic design strategies are presented and compared to predictions from two objective speech quality metrics.

This paper describes usage of MT metrics in choosing the best candidates for MT-based query translation resources in Cross-Language Information Retrieval. Our metrics is METEOR. Language pair of our evaluation is English → German, because METEORmetrics does not offer very many language pairs for comparison. English → German has also available many MT programs that can be used in evaluation. We ...

Research in natural language processing (NLP) has both scientific and technological dimensions. In both cases, it is necessary to evaluate the implemented systems in order to assess the success of a study. This article, grounded in the ISO framework for software evaluation, introduces a typology of NLP systems based on the role of language as input or output data, in order to analyze the centra...

Physical non-invasive security has become crucial for cryptographic modules, which are widely used in pervasive computing. International security evaluation standards, such as U.S. Federal Information Processing Standard (FIPS) 140-3 and Common Criteria (CC) part 3 have added special requirements addressing physical non-invasive security. However, these evaluation standards lack of quantitative...

We consider the evaluation of a business process quality, in particular the evaluation of its robustness. By robustness, we mean robustness w.r.t. the risk of loosing knowledge of persons implied in the business process. We define metrics taking tacit knowledge into account. These metrics are based on the analysis of a social network underlying the process execution. We illustrate these metrics...

Interpretability and discriminative power are the two most basic requirements for an evaluation metric. In this paper, we report the mention identification effect in the B3, CEAF, and BLANC coreference evaluation metrics that makes it impossible to interpret their results properly. The only metric which is insensitive to this flaw is MUC, which, however, is known to be the least discriminative ...