Architecture

​

Lifecycle (shared by every domain)

RESOLVE -> VALIDATE -> PREFLIGHT -> SETUP -> EXECUTE -> COLLECT
        -> SCORE -> ENRICH -> PERSIST -> optional PUBLISH

• RESOLVE — look up the DomainPack, Task and Adapter for a RunSpec.
• VALIDATE — parse and check the RunSpec, target, params and task config (core/validation.py). RESOLVE and VALIDATE failures are UserInputErrors: CLI exit code 2, no record written.
• PREFLIGHT — adapter.preflight(task): credentials, SDK, connectivity and the minimal permissions this benchmark needs on this cloud, before provisioning. A CRITICAL failure produces an invalid record and never enters SETUP. Bypass with --skip-preflight; run standalone with csbench doctor.
• SETUP — adapter.setup(): provision (Terraform) or connect (SDK/HTTP).
• EXECUTE — task.execute(adapter, params): drive the workload, return an ObservationBundle of raw evidence only.
• COLLECT — core/observation.py::collect(): prove the bundle is well formed and canonically encodable.
• SCORE — task.score(bundle): a pure function producing measurements and findings. A scorer failure keeps the observations and records a SCORE error.
• ENRICH — installed ResultEnrichers, each isolated: a failure becomes an ENRICH stage error and never changes status.
• PERSIST — core/store.py: temp file, flush, fsync, atomic rename, plus an emergency dump into the system temp directory (path printed) when the results directory cannot be written.
• PUBLISH — off unless a ResultPublisher is injected. Runs after PERSIST and writes an append-only receipt; it can never rewrite the core record.

​

Result contract (schema 0.2)

• status ∈ completed · failed · invalid · unsupported. There is no boolean ok, no top-level metrics, no top-level evidence_layer and no config_hash.
• environment.execution ∈ simulated · live · unknown marks whether the numbers came from a simulated runtime or a real cloud. It is folded into the environment fingerprint, so simulated and live results never pool together.
• Every measurement carries value, unit and evidence, optionally aggregation, sample_count and notes.
• Every finding carries a stable code, a severity, a summary, its evidence and details.
• Every stage error carries stage, code, type, message and retryable. Tracebacks are never stored in a record; csbench run --debug writes them to <results>/debug/<run_id>.log.
• All digests are full SHA-256 over the canonical JSON encoding (core/canonical.py): UTF-8, sorted keys, no insignificant whitespace, NaN/Infinity rejected.
• extensions["core"] is reserved for the core; a plugin writes under its own name (for example extensions["cb-pricing"]).

​

Run plans and statistics (Phase 1C)

• discards the warmup runs (cold-start / JIT / cache effects are not the steady state you publish), tagging each record’s role under extensions["core"]["run_plan"] so the choice is auditable, never a fingerprint;
• reduces the measured runs to one distribution per measurement (core/statistics.py): numeric → n / mean / stdev / min / max / p50 / p95 / cv, label → distribution / mode / agreement;
• writes a run_plan_aggregate (with its own SHA-256 digest) under results/aggregates/…, which the report loader skips as a summary-of-records.

​

Layers

CLI (csbench)
  └─ Orchestrator (lifecycle state machine)
       ├─ Registry (entry-point domain discovery)
       ├─ DomainPack  → Task(s) + Adapter(s)         [per product category]
       │     └─ Adapter (setup/submit/teardown)      [per (domain, cloud)]
       │           └─ WorkloadEngine (JSONL, any language)  [per load generator]
       ├─ Schema (RunSpec / ResultRecord 0.2 / fingerprints / evidence layer)
       └─ Report (per-dimension matrix + red flags)

​

Plugin contracts

​

Control plane vs. data plane

​

Evidence layers

​

Current domains

• agent-runtime — sessions, tool calling, fault recovery, observability, cost, isolation. All tasks run on local-sim; the full task list and adapter statuses are generated from the registry below (never hand-maintained — see “Maintaining this document”). It spans provisioning, runtime behaviour, tool registration, observability, cost and tenant isolation. Adapters use a managed/transport/mode split so the same code drives the local simulated runtime or, once a runtime-provider plugin is installed, a real cloud; latency-class tasks route through the run_data_plane_probe seam (see “Control plane vs. data plane”); capability probes raise CapabilityNotSupported → recorded as a finding, never a crash.
• Reliability group — platform-as-agent-host stance. The unit under test is the platform as a host for a fixed agent, not the raw invoke API, so the reliability tasks measure whether the “platform + agent” combination recovers when a downstream tool genuinely fails or stalls. T1.3 (fault recovery) / T1.10 (retry storm) / T1.12 (head-of-line) inject faults/latency through the mock tool server’s /fault/config + /latency/config (so they are platform-visible on the real agent→tool HTTP hop, authenticated with the mock token like every other call), give the benchmark agent a pinned retry contract (AGENT_RETRY_POLICY in agent_bundle/lc_agent.py: HTTP 5xx → retry twice, 200 ms backoff, 3 attempts total; 4xx / connection failures → no retry — part of the agent fingerprint, not a tunable knob), and observe by reading the mock’s own call counter (correlation-id bucketed so concurrent requests never cross-count) rather than inferring from the client loop. Each yields a three-state platform attribution — e.g. T1.3 → recovered vs platform_terminated (invoke cut before the agent could recover) vs fail-fast; T1.10 → storm_bounded_by agent / platform / none. T1.2 (state persistence) / T1.11 (concurrent writes) are honestly downgraded to unsupported with an evidence-A finding: FC-based AgentRun has no native session state, so a stateful platform legitimately scores a positive result here — a truthful differentiating negative, not a fabricated pass.
• bigdata-emr — minimal domain pack proving the abstraction generalizes: J1.1 wordcount smoke via the cross-language workload protocol. (This is a small task/adapter surface, not the skeleton AdapterStatus value — its local-process adapter is reference; only its aws-emr adapter is skeleton.)

​

Task & adapter inventory

​

agent-runtime — 27 tasks

​

bigdata-emr — 1 task

​

0.2 Developer Preview readiness

• reference and wired adapters can execute.
• experimental adapters can execute with preview caveats.
• skeleton adapters are discoverable but rejected before preflight.
• Current runnable references are local-sim and local-process. The Aliyun AgentRun (and AWS AgentCore) RuntimeProviders are registered in-tree. The aliyun-agentrun adapter is experimental: it runs end-to-end in mode: mock and its live path has run a full 27-task real-cloud campaign (cn-hangzhou, 2026-08-15 — 25 completed + 2 honestly unsupported). It is not yet promoted to wired (that status is reserved for a stabilized, repeatedly-validated live path); AWS AgentCore and the other clouds remain skeleton.
• Bundled workloads live in clousight_bench.resources.workloads and are resolved with core.resources.reference_workload_path(), so wheel and editable installs use the same files.

• cb-pricing — a fuller / fresher private price feed, consumed by the open pricing enricher via the CLOUSIGHT_PRICING_DATA environment variable (not an entry point);
• cb-dataservice — token-gated private / held-out datasets, registered as a PrivateAssetResolver on the clousight_bench.asset_resolvers entry point, plus a managed object-store upload (placeholder);
• cb-adapters-enterprise — an empty placeholder for future managed-SaaS and private-cloud / 信创 adapters.

​

凭证与上手（便捷层）

• core/credentials.py::resolve_credentials(target, platform) 只探测凭证来源、从不读取/存储密钥值。解析顺序：auth_env（逃生口，显式指定 env 名）→ profile（CLI 命名档）→ 标准 env 变量 → 凭证文件。返回 CredentialResolution(ok, source, identity_hint, remediation)，identity_hint 只含变量名/档名，永不含密钥。
• ProviderAdapter.resolve_credentials() 委托给上面的解析器（跨域，provider 由 target.provider 或平台名前缀推断，如 aliyun-agentrun→aliyun）。真 adapter 运行时仍交给官方 SDK 的默认链。
• 各云凭证链登记在 PROVIDER_CREDENTIALS（aws / aliyun / huawei / volcengine：标准 env、profile env、凭证文件、SDK 模块、文档链接）。
• CLI：csbench init <provider> 生成私有 *.local.yaml + .env.example 并写 .gitignore（配置无密钥）；csbench doctor --config x 分步体检（provider → SDK 可导入？→ 凭证链可解析？→ mock_base_url 可达？localhost 直接判失败），每步给可操作补救提示。

• ProviderAdapter.preflight() -> PreflightReport：默认校验凭证 + provider SDK。跨域可用。
• AgentRuntimeAdapter.preflight(task)：super() 之上，对真云平台（provider 非空）追加 mock_base_url 可达校验 + check_permissions(task)。local-sim（provider 为空、自托管 mock）不加这两项，永远通过。
• orchestrator 在 RESOLVE 之后、SETUP 之前跑 adapter.preflight(task)；任一 CRITICAL 失败即早退（不 provision），落一条 ok=False、metrics.preflight_ok=False、error="preflight failed: ..."、notes= 清单的记录。--skip-preflight 关闭该 gate。

​

最小权限：(benchmark × 云) 矩阵映射

• Task 声明 WHAT：Task.required_permissions: tuple[str, ...] 列出抽象能力令牌（与云无关）。令牌定义在 domains/agent_runtime/permissions.py：session:create / session:state / tool:invoke / tool:register / trace:read / trace:export。当前五维声明：T1.2=session:create,session:state；T1.3=session:create,tool:invoke；T2.1=tool:register；T4.1=session:create,tool:invoke,trace:read；T4.2=session:create,tool:invoke,trace:export。
• Adapter 说 HOW + 验：每个真云 adapter 定义 PERMISSION_MAP（令牌 → 该云的具体最小 IAM/RAM 动作），required_actions(task) 把 task 的令牌解析为该云动作集，_probe_permissions(actions) 做真实鉴权（默认 None=未验证 → WARNING 并列出「本次 benchmark 在该云所需的最小动作」；接真 adapter 覆写为 STS GetCallerIdentity / RAM policy 模拟 / dry-run，返回 (ok, missing) → 缺失即 CRITICAL 阻断）。

​

测评集分发（三层，一套解析机制）

• manifest assets: 段（workload / task）：
  assets:
    - name: tpcds-sf1
      source: remote            # bundled | remote | private
      uri: https://.../tpcds_sf1.tar.zst
      sha256: "…"
      license: TPC-EULA
      version: "1"
  
• core/assets.py：AssetSpec + resolve_asset(spec, base_dir, cache_dir, private_resolver)；WorkloadEngine.resolve_assets() 在 run 前解析全部资产，把 {name: 本地路径} 通过 params["assets"] 暴露给 workload。
• 可复现不泄密：describe() 把资产 identity()（name@version + sha256）折进 config_hash——只存指纹，不存内容；判分键内容永不入 record。
• 判分键抗污染：一个 task 的方法（怎么判）可开源，但其 held-out 键值可声明为 source: private——“怎么判”公开、“标准答案”私有，天然抗背题。
• 扩展点：私有解析器实现 PrivateAssetResolver（name + resolve(spec, cache_dir)），经 clousight_bench.asset_resolvers 注册；开源核心不带任何实现。clousight-bench-pro 的 cb-dataservice 提供 DataServiceAssetResolver（CLOUSIGHT_BENCH_TOKEN 鉴权 + sha256 校验）。

​

数据契约与扩展点

• metrics（标量判分）— 报告层聚合、比较用的数字。
• series（时序，{name: [[t, value], ...]}）— 高频采样，如逐秒延迟、GPU 利用率。
• artifacts（{"kind", "path"/"uri", "media", "sha256"} 指针）— OTel trace、日志包等大文件，record.json 里只存指纹，不存内容。

{"type": "sample", "series": "latency_ms", "t": 1.0, "value": 87.2}
{"type": "artifact", "kind": "otel_trace", "path": "trace.json", "media": "application/json"}
{"type": "result", "ok": true}

run_id | domain | task_id | platform | config_hash | series | t | value | unit

​

Maintaining this document

• Generated blocks live between {/* BEGIN generated:… */} / {/* END generated:… */} markers (currently the task & adapter inventory). They are rendered from the domain registry — the same source as csbench list --json — by scripts/gen_docs.py. Never edit inside the markers by hand. After adding/renaming a task or changing an adapter status, run python scripts/gen_docs.py and commit the result. CI runs python scripts/gen_docs.py --check and tests/test_docs_inventory.py, so a stale block (a hand-typed “28 tasks” that no longer matches the registry) fails the build instead of shipping.
• Prose sections (everything outside the markers) are part of the definition of done for a behaviour change: when a task’s measurement shape, an adapter’s contract, or a lifecycle rule changes, update the paragraph that describes it in the same PR. A doc that drifts a release behind is worse than no doc — reviewers stop trusting it. Keep each edit small and scoped to what changed; the generated blocks absorb the mechanical facts so the prose only has to carry intent.