The Set Query Benchmark

Many of the application systems being designed today, variously known as marketing information, decision support, and management reporting systems, aim to exploit the strategic value of operational data of a commercial enterprise. These applications depart from the row-at-a-time update transaction model of the DebitCredit [1] and TPC benchmarks, and are almost wholly dependent for their performance on what we name "set queries", queries which need to refer to data from a potentially large set of table rows for an answer. The Set Query benchmark chooses a list of "basic" set queries from a review of three major types of strategic data applications: document search, direct marketing, and decision support. In Section 1 of what follows, the data and queries used in the Set Query benchmark are explained and motivated. In Section 2, benchmark results are presented for two leading database products used in large scale operations: IBM's DB2 and Computer Corporation of America's (CCA's) MODEL 204. Surprisingly large performance differences, factors of ten or more for some queries, are observed with respect to I/O, CPU and elapsed time, emphasizing the critical value of benchmarks in this area. In Section 3, a detailed explanation is given of how to generate the data and run the benchmark on an independent platform.