Skip to main content
Clousight Bench is an eval tool for cloud computing. It runs recognized benchmarks — SWE-bench, TPC-DS/H, YCSB, TPC-C, MMLU, GSM8K, HumanEval — unmodified against cloud products and services across five categories (agent runtimes, data warehouses, transactional databases, key-value stores, LLM endpoints), and gives you a number with a verifiable provenance chain: which benchmark, which pinned dataset, which evaluator, on which system-under-test. It never re-scores a benchmark or blends scores. The benchmark’s own harness produces the verdict; the framework adds the reproducibility bookkeeping and the cloud-product dimensions (latency, cost, per-item scores, an optional LLM judge).

​
Quickstart

1

Install

2

See what's installed

Every registered benchmark, evaluator, and platform — plus a copy-pasteable “run one” hint.
3

Run one — offline, no cloud, no key

You get a status: completed record on stdout. ✅ That’s a full run of the four-phase lifecycle (prepare → connect → measure → conclude) over a bundled fixture — no cloud account, no API key. Any run that does not complete prints a one-paragraph self-explanation on stderr — which stage broke, or which gate refused — so you never grep the JSON for the failure.
4

Point it at a real system (config-connect)

Same command shape; now the number comes from a live endpoint you already run. Then csbench serve opens the web viewer at http://127.0.0.1:8787.
Gate a benchmark in CI with a threshold: csbench run … --assert thresholds.yaml exits non-zero if any measurement misses its bound. See pytest & CI.

​
The mental model

Three things, each a plugin discovered by an entry point:
  • A BenchmarkSuite wraps a recognized harness (or any reproducible workload): resolve → prepare → run → mock_artifacts.
  • An Evaluator turns the run’s artifacts into namespaced Measurements (and optional per-item items) — a pure, offline, re-scorable function.
  • The system under test is chosen by --platform + a target: a simulated reference, an already-running service you config-connect to (endpoint + credentials), or a cloud the framework provisions.
You add a benchmark by writing a Suite + Evaluator; you run it with csbench run --benchmark <id>. See Add a benchmark.

​
The reproducibility contract

Every record carries a provenance block (benchmark id, pinned dataset digest, evaluator id, unmodified flag, scaffold) plus three independent fingerprints (benchmark / environment / implementation), so you can tell whether two numbers are even comparable. Every measurement carries a reproducibility_class: official=true measurements are the benchmark’s own verdict under its <suite_id>. namespace (a provenance flag, enforced by csbench conformance); add-on metrics and community evaluators stay in their own space. Results are always per dimension — never a blended score.

​
Next

Add a benchmark

Write a Suite + Evaluator and run it.

Metrics

Compose per-item metrics over any run.

Judges (LLM-as-judge)

Reproducible judge-based scoring.

Architecture

The stage machine, record schema, plugins.

pytest & CI

Assert thresholds in your test suite.

Web viewer

Record browser + trace waterfall.