Architecture

​

Lifecycle (shared by every domain)

(1) PREPARE    RESOLVE -> VALIDATE -> DESCRIBE ... PREFLIGHT ... DESCRIBE
               nothing touched, nothing billed — a bad request stops here
(2) CONNECT    SETUP ............................ TEARDOWN (finally)
               the only window that talks to the cloud, the only one that spends
(3) MEASURE    EXECUTE -> SEAL
               produces observations, never a verdict
    ───────── the cloud is gone after this line ─────────
(4) CONCLUDE   SCORE -> ENRICH -> PERSIST -> optional PUBLISH
               pure, offline, re-scorable; an enricher may add, never overrule

• RESOLVE — look up the DomainPack, benchmark suite (suite:<id>) and Adapter for a RunSpec. Not recorded in run.stages: a RESOLVE failure writes no record at all, so “RESOLVE: ok” would be a constant in every record.
• VALIDATE — parse and check the RunSpec, target, params and task config (core/validation.py). RESOLVE and VALIDATE failures are UserInputErrors: CLI exit code 2, no record written.
• DESCRIBE — assemble the run’s identity, environment and the three fingerprints from what the plugins declare (workload_identity, provenance, environment_facts). It spans the preflight gate: facts that need a live adapter are collected only after PREFLIGHT passes, so DESCRIBE is recorded once, at the end. A plugin that crashes here yields an invalid record and nothing is provisioned. (Before 0.6.0 these failures were filed under VALIDATE, which produced records reading “VALIDATE failed, PREFLIGHT ok”.)
• PREFLIGHT — adapter.preflight(task): credentials, SDK, connectivity and the minimal permissions this benchmark needs on this cloud, before provisioning. A CRITICAL failure produces an invalid record and never enters SETUP. Bypass with --skip-preflight; run standalone with csbench doctor.
• SETUP — adapter.setup(): provision (Terraform) or connect (SDK/HTTP). Both setup() and teardown() are optional no-op hooks — the three cloud phases create → connect → destroy are separable, and a connect-only run (config-connect to an already-running service via target.endpoint / credentials_ref) uses only connect. Whether a run provisions is an explicit adapter capability, provisions_resources() (default: real cloud executed live; False for *-endpoint config-connect adapters and simulators, or inherit ProvisionedCloudAdapter to always declare True). The provisioned-cloud machinery — the live-run confirmation gate, the cost budget/ledger, and resource reconciliation/reaping — runs only when provisions_resources() is True; a connect-only run skips all of it.
• EXECUTE — task.execute(adapter, params): drive the workload, return an ObservationBundle of raw observations only.
• SEAL — core/observation.py::seal(): prove the bundle is well formed and canonically encodable. It fetches nothing from the cloud — EXECUTE already returned the bundle; this is the seal before scoring, and it is a stage of its own so “the suite crashed” and “the suite returned something unencodable” are different, attributable failures. Named COLLECT before 0.6.0; the old name stays valid in STAGES so records written by 0.5.x still load.
• SCORE — task.score(bundle): a pure function producing measurements and findings. A scorer failure keeps the observations and records a SCORE error.
• ENRICH — installed ResultEnrichers, each isolated: a failure becomes an ENRICH stage error and never changes status.
• PERSIST — core/store.py: temp file, flush, fsync, atomic rename, plus an emergency dump into the system temp directory (path printed) when the results directory cannot be written.
• PUBLISH — off unless a ResultPublisher is injected. Runs after PERSIST and writes an append-only receipt; it can never rewrite the core record.

​

Result contract (schema 0.4)

• items (schema 0.4, optional) is the per-item substrate: a list of ItemResults (item_id, group, input/output/reference, per-item scores with a 4-state status ∈ ok·fail·skip·error, usage). The scalar measurements are their aggregation (core/aggregate.py — mean/ratio/ geomean/percentile + Wilson/normal confidence intervals + per-group breakdowns). Empty for suites not yet migrated to per-item scoring. Capped by CSBENCH_MAX_PERSISTED_ITEMS (default 1000); truncation is recorded under extensions.core.items_meta, never silent.
• status ∈ completed · failed · invalid · unsupported for any record execute() hands back. A fifth value, interrupted, exists only for Ctrl-C: teardown runs, the record is persisted, and the KeyboardInterrupt is then re-raised — so you find that status on disk, never as a return value. There is no boolean ok, no top-level metrics, no top-level evidence_layer and no config_hash.
• environment.execution ∈ simulated · live · unknown marks whether the numbers came from a simulated runtime or a real cloud. It is folded into the environment fingerprint, so simulated and live results never pool together.
• Every measurement carries value, unit and official, optionally reproducibility_class, aggregation, sample_count, notes and ci.
• Judge-based metrics (reproducibility_class="judge-based") are real via the judge base (core/judge.py): a JudgeModel (config-connected EndpointJudge, SSRF-guarded) is asked for a CATEGORICAL verdict and fixed arithmetic maps it to a score — G-Eval’s non-reproducible top-logprob weighting is deliberately rejected. judge_emit gets structured output from any judge (native JSON-schema mode, else prompt + brace-slice/trailing-comma repair). The response-quality metric is the reference: a rubric judge with optional self-consistency; it skips cleanly when no judge is configured (so CI never calls a live judge) and records the judge model id + verdict in each item score. Run it by opting the metric in and config-connecting a judge: params: {extra_metrics: [response_quality], judge: {provider: openai-compatible, endpoint: ..., model: ..., credentials_ref: env:KEY, cache: .judge_cache.json}} — a judge-configured run does network I/O in scoring and is not offline-re-scorable, by design. A cache path wraps the judge in a content-addressed CachingJudge so a re-run reuses verdicts and never re-pays the LLM calls (only judge scoring is cached — never environmental numbers).
• Golden ↔ executed split: the benchmark spec is the versioned DatasetHandle (version + digest, pinned in-repo) plus each item’s reference (golden); the run’s output is the executed side. No separate Golden subsystem — the per-item substrate already carries both.
• Every finding carries a stable code, a severity, a summary and details.
• Every stage error carries stage, code, type, message and retryable. Tracebacks are never stored in a record; csbench run --debug writes them to <results>/debug/<run_id>.log.
• All digests are full SHA-256 over the canonical JSON encoding (core/canonical.py): UTF-8, sorted keys, no insignificant whitespace, NaN/Infinity rejected.
• extensions["core"] is reserved for the core; a plugin writes under its own name (for example extensions["cb-pricing"]).

​

Run plans and statistics (Phase 1C)

• discards the warmup runs (cold-start / JIT / cache effects are not the steady state you publish), tagging each record’s role under extensions["core"]["run_plan"] so the choice is auditable, never a fingerprint;
• reduces the measured runs to one distribution per measurement (core/statistics.py): numeric → n / mean / stdev / min / max / p50 / p95 / cv, label → distribution / mode / agreement;
• writes a run_plan_aggregate (with its own SHA-256 digest) under results/aggregates/…, which the report loader skips as a summary-of-records.

​

Layers

CLI (csbench)
  └─ Orchestrator (lifecycle state machine)          [layer ① single-run eval]
       ├─ Registry (entry-point discovery)
       ├─ BenchmarkSuite + Evaluator (+ Metric/Judge)  [the benchmark contract]
       │     └─ SuiteRunner (internal: observation packing + pure scoring)
       ├─ DomainPack → Adapter(s)                    [layer ② cloud lifecycle]
       │     └─ Adapter (setup/submit/teardown)      [per (domain, cloud)]
       │           └─ WorkloadEngine (JSONL, any language)  [per load generator]
       ├─ Schema (RunSpec / ResultRecord 0.4 / fingerprints / provenance)
       └─ Viewer (csbench serve: records, provenance, trace)

clousight_bench.ops (run-plans, analytics/rollups)   [layer ③ campaign ops]
  └─ imports core; core never imports ops

​

Plugin contracts

​

Control plane vs. data plane

​

Evidence layers

​

The benchmark contract (BenchmarkSuite + Evaluator)

• BenchmarkSuite drives an upstream harness unmodified and returns opaque RawArtifacts (a directory + manifest of named files). Core treats the artifacts as a black box; only the paired evaluator understands them.
• Evaluator reads RawArtifacts as a pure function (no cloud, no credentials) and returns dict[str, Measurement] where every key is namespaced "<suite_id>.<metric_name>" (official evaluators) or "<evaluator_id>.<metric_name>" (custom evaluators).
• Namespace / official invariant: conformance.check_evaluator(evaluator, suite_id, measurements) verifies that official evaluators emit only official=True measurements under the suite namespace, and custom evaluators emit only official=False measurements under their own id — and that custom evaluators never squat a registered suite’s namespace. Returns list[CheckResult]; run it from the CLI with csbench conformance --suite <suite_id> (exercises every registered evaluator over the suite’s mock artifacts), or compose with run_conformance for CI gates.
• SuiteRunner + Provenance (internal runner, not a plugin contract): threads provenance metadata (suite_id, suite_version, evaluator_id, unmodified flag) through every TaskResult and folds it into the benchmark fingerprint, so records from different suite versions or evaluators are never pooled.
• SWE-bench pilot (slice 1): SweBenchSuite (registered entry point swe-bench) + OfficialSweEvaluator (emits swe-bench.resolved and swe-bench.cost_per_resolved) + bundled fixtures for offline / CI use. The real Docker-backed run() path requires the [swebench] extra and is gated behind @pytest.mark.slow.
• SWE-bench Lite (variant): the 300-instance curated subset ships as SweBenchLiteSuite (entry point swe-bench-lite) + OfficialSweLiteEvaluator (emits swe-bench-lite.resolved / swe-bench-lite.cost_per_resolved). It is a thin subclass of the flagship — identical harness, report shape, and agent-runtime plumbing; only the pinned HuggingFace split (princeton-nlp/SWE-bench_Lite), the bundled real fixtures, and the namespace differ. This is the parametrization seam future variants (Multimodal) reuse: a variant overrides the suite_id / suite_version / fixtures_dir / dataset_name class attributes and inherits everything else.
• SWE-bench Multimodal (variant): image-augmented issues from real JavaScript / visual-frontend repos, shipped as SweBenchMultimodalSuite (entry point swe-bench-multimodal) + OfficialSweMultimodalEvaluator (emits swe-bench-multimodal.resolved / .cost_per_resolved). Same thin-subclass seam as Lite, plus a 7th fixture field image_assets (image URLs) threaded to the agent by _load_instance. It binds the dev split on purpose: the 510-instance test leaderboard split has its gold patches held out (scored via the hosted sb-cli service), whereas the 102-instance dev split ships real gold patches and is locally Docker-runnable — matching how Verified/Lite work. Wiring the hosted test-split path is a documented future (gated) addition.

​

Current domains

• agent-runtime — the cloud-infra shell for managed agent runtimes: adapters, runtime providers, probes, reaper. Since the suite-first pivot it registers no self-designed tasks — benchmark runs arrive as suite:<id> jobs through the benchmark-suite contract. Adapters use a managed/transport/mode split so the same code drives the local simulated runtime or, once a runtime-provider plugin is installed, a real cloud; capability probes raise CapabilityNotSupported → recorded as a finding, never a crash. The adapter statuses below are generated from the registry (never hand-maintained — see “Maintaining this document”).

​

Benchmark & adapter inventory

​

Benchmarks — 10 suites

​

agent-runtime

​

data-warehouse

​

key-value

​

llm

​

transactional-db

​

Developer Preview readiness

• reference and wired adapters can execute.
• experimental adapters can execute with preview caveats.
• skeleton adapters are discoverable but rejected before preflight.
• All five domains ship a runnable reference adapter: local-sim, duckdb-local, ycsb-local, benchbase-local and llm-mock. The Aliyun AgentRun (and AWS AgentCore) RuntimeProviders are registered in-tree. The aliyun-agentrun adapter is experimental: it runs end-to-end in mode: mock and its live path ran a real-cloud campaign (cn-hangzhou, 2026-08-15 — 25 completed + 2 honestly unsupported). It is not yet promoted to wired (that status is reserved for a stabilized, repeatedly-validated live path); AWS AgentCore and the other clouds remain skeleton.
• Bundled workloads live in clousight_bench.resources.workloads and are resolved with core.resources.reference_workload_path(), so wheel and editable installs use the same files.

• cb-pricing — a fuller / fresher private price feed, consumed by the open pricing enricher via the CLOUSIGHT_PRICING_DATA environment variable (not an entry point);
• cb-dataservice — token-gated private / held-out datasets, registered as a PrivateAssetResolver on the clousight_bench.asset_resolvers entry point, plus a managed object-store upload (placeholder);
• cb-adapters-enterprise — an empty placeholder for future managed-SaaS and private-cloud / domestic-stack adapters.

​

Credentials & onboarding (convenience layer)

• core/credentials.py::resolve_credentials(target, platform) only probes the credential source; it never reads or stores key values. Resolution order: auth_env (escape hatch — explicitly named env var) → profile (named CLI profile) → standard env vars → credential file. Returns CredentialResolution(ok, source, identity_hint, remediation); identity_hint contains only the variable name or profile name, never the key material.
• ProviderAdapter.resolve_credentials() delegates to the resolver above (cross-domain; provider is inferred from target.provider or the platform-name prefix, e.g. aliyun-agentrun → aliyun). The real adapter runtime still hands off to the official SDK’s default chain.
• Each provider’s credential chain is registered in PROVIDER_CREDENTIALS (aws / aliyun / huawei / volcengine: standard env vars, profile env, credential file, SDK module, documentation link).
• CLI: csbench init <provider> generates a private *.local.yaml + .env.example and writes .gitignore (configuration contains no secrets); csbench doctor --config x runs a step-by-step health check (provider → SDK importable? → credential chain resolvable? → mock_base_url reachable? localhost fails immediately), plus target-connectivity probes for config-connect runs: TCP reachability of the endpoint, a Redis RESP PING, and java -version gates (BenchBase needs Java ≥ 17, YCSB ≥ 11) — all SSRF-guarded; mock targets skip the probes. Each step provides an actionable remediation hint.

• ProviderAdapter.preflight() -> PreflightReport: validates credentials + provider SDK by default. Usable across domains.
• AgentRuntimeAdapter.preflight(task): on top of super(), appends a mock_base_url reachability check + check_permissions(task) for real cloud platforms (non-empty provider). local-sim (empty provider, self-hosted mock) skips both additions and always passes.
• The orchestrator runs adapter.preflight(task) after RESOLVE and before SETUP; any CRITICAL failure triggers an early exit (no provisioning): the record gets a StageError(stage="PREFLIGHT", code="preflight_failed") plus a Finding(code="core.preflight_failed", severity="critical") whose details.checks lists every check line, and the record’s status is invalid. --skip-preflight disables this gate.

​

Least privilege: the (benchmark × cloud) matrix

• The benchmark declares WHAT: a suite’s required_permissions: tuple[str, ...] lists abstract capability tokens (cloud-agnostic). Tokens are defined in domains/agent_runtime/permissions.py: session:create / session:state / tool:invoke / tool:register / trace:read / trace:export.
• Adapter says HOW + verifies: each real-cloud adapter defines a PERMISSION_MAP (token → the concrete minimum IAM/RAM actions for that cloud). required_actions(task) resolves the task’s tokens into the cloud-specific action set; _probe_permissions(actions) performs the actual authorization check (the default implementation returns None = unverified → emits a WARNING listing the minimum actions required for this benchmark run on this cloud; real adapter overrides call STS GetCallerIdentity / RAM policy simulation / dry-run and return (ok, missing) — any missing action is a CRITICAL blocker).

​

Dataset distribution (three tiers, one resolver)

• assets: section in the manifest (workload / task):
  assets:
    - name: tpcds-sf1
      source: remote            # bundled | remote | private
      uri: https://.../tpcds_sf1.tar.zst
      sha256: "…"
      license: TPC-EULA
      version: "1"
  
• core/assets.py: AssetSpec + resolve_asset(spec, base_dir, cache_dir, private_resolver); WorkloadEngine.resolve_assets() resolves all assets before a run and exposes {name: local_path} to the workload via params["assets"].
• Reproducible without leaking secrets: describe() folds each asset’s identity() (name@version + sha256) into the config_hash — only the fingerprint is stored, never the content; scoring-key content never enters the record.
• Scoring-key contamination resistance: a task’s methodology (how to score) can be open-sourced while its held-out key values are declared source: private — “how to score” is public, “the answer key” is private, which naturally prevents benchmark leakage.
• Extension point: private resolvers implement PrivateAssetResolver (name + resolve(spec, cache_dir)) and register via clousight_bench.asset_resolvers; the open core ships no implementation. The cb-dataservice component of clousight-bench-pro provides DataServiceAssetResolver (CLOUSIGHT_BENCH_TOKEN authentication + sha256 verification).

​

Data contract & extension points

• measurements (scalar scores) — numbers used by the report layer for aggregation and comparison (there is no top-level metrics field).
• series (time series, {name: [[t, value], ...]}) — high-frequency samples such as per-second latency or GPU utilization.
• artifacts ({"kind", "path"/"uri", "media", "sha256"} pointers) — large files such as OTel traces and log bundles; only the fingerprint is stored in record.json, not the content.

{"type": "sample", "series": "latency_ms", "t": 1.0, "value": 87.2}
{"type": "artifact", "kind": "otel_trace", "path": "trace.json", "media": "application/json"}
{"type": "result", "ok": true}

run_id | domain | task_id | platform | benchmark_fingerprint | series | t | value | unit

​

Maintaining this document

• Generated blocks live between {/* BEGIN generated:… */} / {/* END generated:… */} markers (currently the task & adapter inventory). They are rendered from the domain registry — the same source as csbench list --json — by scripts/gen_docs.py. Never edit inside the markers by hand. After adding/renaming a task or changing an adapter status, run python scripts/gen_docs.py and commit the result. CI runs python scripts/gen_docs.py --check and tests/test_docs_inventory.py, so a stale block (a hand-typed “28 tasks” that no longer matches the registry) fails the build instead of shipping.
• Prose sections (everything outside the markers) are part of the definition of done for a behaviour change: when a task’s measurement shape, an adapter’s contract, or a lifecycle rule changes, update the paragraph that describes it in the same PR. A doc that drifts a release behind is worse than no doc — reviewers stop trusting it. Keep each edit small and scoped to what changed; the generated blocks absorb the mechanical facts so the prose only has to carry intent.