IT organizations use strategic IT benchmarking (SITBM) to revise IT strategies or perform internal marketing. Despite benchmarking’s long tradition, many SITBM initiatives do not reveal the desired outcomes. The vast body of knowledge on benchmarking and IT management does not help overcome the challenges of successful SITBM. We therefore synthesize existing research on IT management and benchm...

In this paper, we present an initial framework of evaluation and benchmarking of reasoners deployed within the LarKC platform, a platform for massive distributed incomplete reasoning that will remove the scalability barriers of currently existing reasoning systems for the Semantic Web. We discuss the evaluation methods, measures, benchmarks, and performance targets for the plug-ins to be develo...

There are numerous approaches towards quantifying the performance of NoSQL datastores with respect to dimensions that are notoriously hard to capture such as staleness or consistency in general. Many of these approaches, though, are built on assumptions regarding the underlying infrastructure or the test scenario and may lead to invalid results, if those assumptions do not hold. As a consequenc...

The increasing overall global competition forces requirements for companies to improve their business performance continuously in every sector. The utilization of external resources, including a supplier network, has become one of the most critical development areas of business, needing a lot of attention. In this paper, the performance of buyer-supplier relatio nships will be enhanced by using...

Nowadays, globalization and competitiveness are major challenges in almost all organizations and managers are seeking modern approaches to gain more competitive advantages. Considering these circumstances, issues such as strategic management, benchmarking and balanced score card (BSC) become important. In this paper, benchmarking and its role in strategic management have been demonstrated. BSC ...

Computer benchmarks are standard tools that allow evaluating and comparing different systems or components according to specific characteristics (performance, dependability, security, etc). Resilience encompasses all attributes of the quality of ‘working well in a changing world that includes faults, failures, errors and attacks’. This way, resilience benchmarking merges concepts from performan...

The dynamical and uncertain environments of domestic service robots, which include humans, require rethinking of the benchmarking principles for testing these robots. Since 2006 RoboCup@Home has used statistical procedures to track and steer the progress of domestic service robots. This paper explains the procedures and shows outcomes of these international benchmarking efforts. Although aspect...

Effective benchmarking requires standards for measuring performance across a broad range of organizations. Industry benchmarking measures the relative performance levels of similar operations, but gathering sufficient data to robustly characterize best performance is the primary hurdle to more widespread implementation. This paper describes the development of an innovative, large-scale benchmar...

The design of the information-processing architecture used to develop an intelligent robot plays a significant role in the behaviour of the final system. In this paper we discuss the possibilities for benchmarking the influence of architecture designs on intelligent robots. We separate this problem into two sub-problems: benchmarking the architecture design and benchmarking the implementation o...

This survey report highlights the main topics associated with availability benchmarking of computing and data storage systems. Compared to performance benchmarking, availability is still in its infancy. A widely adopted benchmarking framework has yet to be designed. This is partly due to the complexity of such a task and the problems associated with it. Numerous factors have to be taken into ac...