TPC-H suite (data-warehouse)

TPC-H runs on the data-warehouse domain’s duckdb-local reference platform (a provider-less, single-node engine, the sibling of the tpc-ds suite). It has two modes selected by params.mode:

reference (default) — run the 22 queries once (single stream), digest each result, emit queries.json + summary.json. Cheap CI/offline path, scored by official-tpch-evaluator.
official — the full official pipeline (Load → Power incl. RF1/RF2 → multi-stream Throughput → ACID), emitting official.json scored by official-tpch-qphh-evaluator into the official QphH@Size composite.

Official-mode numbers reproduce the official TPC-H formulas but are unaudited: no TPC membership, no audit, no priced full-disclosure report. The RF1/RF2 refresh set is clousight-generated. Every score says so in its notes.

Run reference mode (no cloud account)

# offline mock path — no DuckDB needed, exercises the whole stage machine:
printf 'target:\n  mode: mock\n' > mock.yaml
csbench run --domain data-warehouse --benchmark tpc-h \
  --platform duckdb-local --config mock.yaml

# real DuckDB SF1 run (needs the [tpch] extra: pip install clousight-bench[tpch]):
printf 'target:\n  mode: runtime\nparams:\n  scale_factor: 1\n  query_ids: [1, 6, 14]\n' > real.yaml
csbench run --domain data-warehouse --benchmark tpc-h \
  --platform duckdb-local --config real.yaml

Run official mode (QphH@Size)

Official mode requires selecting its evaluator (both evaluators support tpc-h; the reference one is chosen by default):

# offline mock official pipeline:
printf 'target:\n  mode: mock\nparams:\n  mode: official\n  evaluator: official-tpch-qphh-evaluator\n' > off-mock.yaml
csbench run --domain data-warehouse --benchmark tpc-h \
  --platform duckdb-local --config off-mock.yaml

# real DuckDB official run (needs the [tpch] extra):
printf 'target:\n  mode: runtime\nparams:\n  mode: official\n  scale_factor: 1\n  streams: 2\n  evaluator: official-tpch-qphh-evaluator\n' > off-real.yaml
csbench run --domain data-warehouse --benchmark tpc-h \
  --platform duckdb-local --config off-real.yaml

Scaling up

params.scale_factor sets the size (QphH is reported @Size), and params.streams sets the number of concurrent Throughput query streams. It defaults to the official minimum for the scale factor (SF1 → 2, SF10 → 3, SF30 → 4, SF100 → 5, …). All three composites scale with SF, and Throughput@Size grows with the stream count. The query-stream permutations come from params.query_order:

`query_order`	Ordering	Use when
`official` (default)	the official Appendix A table — comparable to published runs. We bundle streams 0–2 and never fabricate the rest; supply the full table yourself via `params.query_order_file` (its sha folds into the dataset digest)	any `S` your table covers
`generated`	a deterministic, reproducible clousight-generated ordering (distinct permutation per stream)	any `S` — needed for SF ≥ 10 until the Appendix A table is extended

official fails loudly if you ask for more streams than the shipped Appendix A table covers — it never silently substitutes an unofficial order. Use query_order: generated to scale past S = 2 (each stream still runs all 22 queries, so the throughput metric is correct; only strict comparability to a published run needs the exact Appendix A sequences). The chosen ordering is recorded in official.json as ordering_source and folded into the run’s dataset digest.

# SF10 official-minimum 3 streams, generated ordering:
printf 'target:\n  mode: runtime\nparams:\n  mode: official\n  scale_factor: 10\n  streams: 3\n  query_order: generated\n  evaluator: official-tpch-qphh-evaluator\n' > sf10.yaml
csbench run --domain data-warehouse --benchmark tpc-h --platform duckdb-local --config sf10.yaml

What official mode measures

The official-tpch-qphh-evaluator emits these tpc-h.* measurements:

Measurement	Meaning	reproducibility_class
`tpc-h.qphh_at_size`	QphH@Size = `sqrt(Power@Size · Throughput@Size)`	`environmental`
`tpc-h.power_at_size`	`Power@Size` = `(3600 / geomean(24 intervals)) · SF`	`environmental`
`tpc-h.throughput_at_size`	`Throughput@Size` = `(S · 22 / Ts) · 3600 · SF`	`environmental`
`tpc-h.load_time_s`	data-load wall-clock	`environmental`
`tpc-h.acid_atomicity` / `_consistency` / `_isolation`	`1.0` pass / `0.0` fail	`deterministic`

Official mode makes no correctness claim: the Power test’s queries run AFTER RF1 has refreshed the data, so their results legitimately differ from the pristine-data references — comparing them would mislabel correct behavior as failure. Correctness (verified against the official answers) rides the reference mode. ACID Durability is reported as n/a (no crash/recovery harness on embedded DuckDB) and therefore omitted from the measurement set. ACID is an independent pass/fail gate — it is not part of the QphH@Size composite. All carry official=True — a provenance flag (“emitted by the suite’s recognized evaluator”), not an audit claim; drift is carried by reproducibility_class.

Correctness: verified official answers

The pinned references for SF 1, 0.1 and 0.01 were each verified cell-by-cell against DuckDB’s official tpch_answers() answer set at capture time (all 22 queries, numeric cells at the shared 2-dp rounding, order-insensitive rows) — queries_passed at those scale factors says so in its notes. Other scale factors carry no correctness claim until you capture a reference (scripts/capture_tpch_reference.py --sf <sf>); still not an audited TPC result.

Price/performance (price/QphH)

Supply your system price through the pricing feed and the pricing enricher emits the official-style composite (unaudited — the price is your own input, e.g. a 3-year TCO or a monthly subscription):

// my-feed.json — point CLOUSIGHT_PRICING_DATA at it
{
  "currency": "USD",
  "prices": [],
  "system_prices": [
    {"perf_metric": "tpc-h.qphh_at_size", "price": 120000.0,
     "basis": "3yr-tco", "provider": "aliyun", "source": "operator quote"}
  ]
}

A record whose tpc-h.qphh_at_size matches (and provider/region filters, when given) gains extensions.pricing.price_performance[] with price_per_unit_perf = price / QphH. The bundled seed ships no system prices — a price/perf number never comes from invented data, and it can never change the verdict (enrichers are additive by contract).

Out of scope (today)

Real-SUT config-connect engines (duckdb-local only), and extending the official Appendix A permutation table beyond streams 0–2 (use query_order: generated to scale in the meantime). Very large scale factors are bounded by single-node DuckDB memory/disk. Durability crash-recovery is not tested on embedded DuckDB.

Get started

Core concepts

Build a benchmark

Benchmarks

Operate

Reference

TPC-H suite (data-warehouse)

Run reference mode (no cloud account)

Run official mode (QphH@Size)

Scaling up

What official mode measures

Correctness: verified official answers

Price/performance (price/QphH)

Out of scope (today)

​Run reference mode (no cloud account)

​Run official mode (QphH@Size)

​Scaling up

​What official mode measures

​Correctness: verified official answers

​Price/performance (price/QphH)

​Out of scope (today)

Run reference mode (no cloud account)

Run official mode (QphH@Size)

Scaling up

What official mode measures

Correctness: verified official answers

Price/performance (price/QphH)

Out of scope (today)