Data generation for application-specific benchmarking

Data Generation for Application-Specific Benchmarking

The Transaction Processing Council (TPC) has played a pivotal role in the database industry’s growth over the last twenty-five years. However, its handful of domain-specific benchmarks are increasingly irrelevant to the multitude of data-centric applications, and its top-down process is slow. This mismatch calls for a paradigm shift to a bottomup community effort to develop tools for applicatio...

Application - Specific Benchmarking

This thesis introduces a novel approach to performance evaluation, called application-specific benchmarking, and presents techniques for designing and constructing meaningful benchmarks. A traditional benchmark usually includes a fixed set of programs that are run on different systems to produce a single figure of merit, which is then used to rank system performance. This approach often overloo...

The Case for Application-Specific Benchmarking

Most performance analysis today uses either microbenchmarks or standard macrobenchmarks (e.g., SPEC, LADDIS, the Andrew benchmark). However, the results of such benchmarks provide little information to indicate how well a particular system will handle a particular application. Such results are, at best, useless and, at worst, misleading. In this paper, we argue for an application-directed appro...

Data Generation for Stream Reasoning Benchmarking

Second Generation Benchmarking and Application Oriented Evaluation

Digital image watermarking techniques for copyright protection have become increasingly robust. The best algorithms perform well against the now standard benchmark tests included in the Stirmark package. However the stirmark tests are limited since in general they do not properly model the watermarking process and consequently are limited in their potential to removing the best watermarks. Here...