TPC-DS suite (data-warehouse)

TPC-DS is Clousight Bench’s second benchmark suite and the first on the data-warehouse domain — the benchmark_suite / evaluator contract generalized beyond agent suites (SWE-bench) to OLAP/data. The first platform is duckdb-local: a provider-less, single-node reference engine (the analog of agent-runtime’s local-sim) that runs entirely offline.

Run it (no cloud account)

# offline mock path — no DuckDB needed, exercises the whole stage machine:
printf 'target:\n  mode: mock\n' > mock.yaml
csbench run --domain data-warehouse --benchmark tpc-ds \
  --platform duckdb-local --config mock.yaml

# real DuckDB SF1 run (needs the [tpcds] extra: pip install clousight-bench[tpcds]):
printf 'target:\n  mode: runtime\nparams:\n  scale_factor: 1\n  query_ids: [3, 7, 42]\n' > real.yaml
csbench run --domain data-warehouse --benchmark tpc-ds \
  --platform duckdb-local --config real.yaml

params.scale_factor defaults to 1; params.query_ids defaults to all 99 queries. Omit query_ids to run the full set (~1.3s at SF1 after a one-time ~6s data generation).

What it measures

The official-tpcds-evaluator emits three tpc-ds.* measurements:

Measurement	Meaning	reproducibility_class
`tpc-ds.queries_passed`	ratio of queries whose result matches the SF-keyed pinned reference (`sf<sf>_digests.json`; only SF1 is shipped — omitted at other SFs)	`deterministic`
`tpc-ds.geomean_latency_ms`	geometric mean of per-query wall-time	`environmental`
`tpc-ds.total_runtime_ms`	sum of per-query wall-time	`environmental`

All carry official=True — in this framework that is a provenance flag (“emitted by the suite’s recognized evaluator”), not an audit claim; whether a number drifts is carried by reproducibility_class.

Correctness: a pinned SF1 reference

DuckDB’s tpcds_answers() is unusable (no parameters, defaults to SF10), so correctness compares each query’s normalized result digest against a pinned SF1 reference captured once against the pinned DuckDB + extension version (suites/tpc_ds/fixtures/reference/sf1_digests.json, produced by scripts/capture_tpcds_reference.py). This is a deterministic reproducibility/regression check against a recognized implementation — not an externally-audited TPC answer — and is emitted only at scale factor 1 (the reference is SF1-only). The digest normalization rounds numerics and sorts rows so it is stable across platforms; a real run reproduces the committed reference in CI.

Deliberate scope (slice 1)

Correctness is SF1-only; higher scale factors report performance only.
An audited QphDS number is never claimed. The unaudited official-formula composite exists: params.mode: official runs the full sequence (load + power
- concurrent throughput streams + data maintenance + ACID gate) and computes QphDS@SF — see the Official mode section below.
Big-data scale and distributed engines (Spark on EMR, cloud data warehouses) are a later phase, added as additional data-warehouse platforms behind the same suite.

Sibling: TPC-H

TPC-H is the second OLAP suite on the same data-warehouse domain and duckdb-local platform — proof that a new suite drops onto the contract cheaply. It shares the shape of TPC-DS (offline mock + real DuckDB, honest per-query latency), just with DuckDB’s tpch extension (dbgen + 22 queries via PRAGMA tpch(n)) — and goes further on correctness and composites: SF-keyed verified references at SF 1 / 0.1 / 0.01 (each verified cell-by-cell against DuckDB’s official tpch_answers() at capture time) and its own official-formula QphH@Size mode. See TPC-H suite.

csbench run --domain data-warehouse --benchmark tpc-h \
  --platform duckdb-local --config mock.yaml   # or a real.yaml with mode: runtime

Measurements are tpc-h.queries_passed / tpc-h.geomean_latency_ms / tpc-h.total_runtime_ms. Needs the [tpch] extra for the real path. Unlike TPC-DS, DuckDB ships usable tpch_answers() official answers, so TPC-H correctness is verified, not merely pinned: scripts/capture_tpch_reference.py verifies every reference entry against them at capture time and stamps it verified_official — 22/22 verified at each of the three shipped scale factors (1, 0.1, 0.01).

Official mode (QphDS@SF)

params.mode: official runs the TPC-DS official sequence — Load → Power (99 queries, single stream) → Throughput Test 1 → Data Maintenance 1 → Throughput Test 2 → Data Maintenance 2 → ACID gate — and the official-tpcds-qphds-evaluator computes the official composite:

QphDS@SF = ⌊ SF · Q / ⁴√(T_PT · T_TT · T_DM · T_LD) ⌋

printf 'target:\n  mode: runtime\nparams:\n  mode: official\n  scale_factor: 1\n  evaluator: official-tpcds-qphds-evaluator\n' > ds-official.yaml
csbench run --domain data-warehouse --benchmark tpc-ds --platform duckdb-local --config ds-official.yaml

The official-tpcds-qphds-evaluator emits:

Measurement	Meaning	reproducibility_class
`tpc-ds.qphds_at_size`	the QphDS@SF composite (unaudited caveats in its `notes`)	`environmental`
`tpc-ds.load_time_s`	Load Test wall time	`environmental`
`tpc-ds.power_test_s`	Power Test — sum of per-query intervals	`environmental`
`tpc-ds.throughput_test_s`	Throughput Tests 1 + 2 combined	`environmental`
`tpc-ds.maintenance_test_s`	Data Maintenance 1 + 2 combined	`environmental`
`tpc-ds.acid_atomicity`, `tpc-ds.acid_isolation`	ACID probe pass/fail (consistency and durability are `n/a` → omitted)	`deterministic`
`tpc-ds.queries_passed`	SF1 only — pinned-reference correctness over the Power stream	`deterministic`

params.streams defaults to the official minimum (4). The run also emits its OTel-native trace (trajectory.jsonl) — the viewer waterfall shows the power queries, the concurrent throughput lanes of both tests, and the DM windows.

Unaudited, with honestly-declared substitutions: DuckDB’s tpcds extension has no LF_* maintenance functions, so data maintenance is a clousight-generated insert+delete round-trip on store_sales; no official permutation table is bundled, so the query ordering is always the deterministic clousight generated one (recorded as ordering_source and folded into the dataset digest); ACID probes atomicity/isolation generically — consistency and durability are n/a. Every composite carries these caveats in its notes.

Get started

Core concepts

Build a benchmark

Benchmarks

Operate

Reference

TPC-DS suite (data-warehouse)

Run it (no cloud account)

What it measures

Correctness: a pinned SF1 reference

Deliberate scope (slice 1)

Sibling: TPC-H

Official mode (QphDS@SF)

​Run it (no cloud account)

​What it measures

​Correctness: a pinned SF1 reference

​Deliberate scope (slice 1)

​Sibling: TPC-H

​Official mode (QphDS@SF)

Run it (no cloud account)

What it measures

Correctness: a pinned SF1 reference

Deliberate scope (slice 1)

Sibling: TPC-H

Official mode (QphDS@SF)