> ## Documentation Index
> Fetch the complete documentation index at: https://docs.clousight.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Add a benchmark

> Write a BenchmarkSuite + Evaluator, register two entry points, run it with csbench run --benchmark.

Adding a benchmark is the primary way to extend Clousight Bench — for open-source
*and* commercial suites (a commercial suite ships the same entry points from a
private package). You implement two classes and register two entry points; no core
change.

<Tip>
  Copy an existing self-contained suite as your template:
  `src/clousight_bench/suites/gsm8k/` (bundled fixture + evaluator + a tiny suite).
  SWE-bench is the most complete example but drags in Docker/HF — start from GSM8K.
</Tip>

## The two contracts

* **`BenchmarkSuite`** produces raw artifacts: `resolve` (pick the dataset,
  offline) → `prepare` (connect/provision) → `run` (produce a `RawArtifacts`
  dir+manifest) → `teardown`; plus `mock_artifacts` (the offline path CI uses).
* **`Evaluator`** scores those artifacts into namespaced `Measurement`s — a pure,
  offline function (no cloud, no credentials), so a stored run is re-scorable.

Import both from the stable surface — `clousight_bench` (or `clousight_bench.api`),
never `clousight_bench.core.*`.

## A minimal suite

```python theme={null}
# my_pkg/echo_suite.py
from clousight_bench import (
    BenchmarkSuite, Evaluator, DatasetHandle, EnvHandle, RawArtifacts,
    Target, DriverContext, Measurement,
)
import json, tempfile
from pathlib import Path


class EchoSuite(BenchmarkSuite):
    suite_id = "echo"                 # == entry-point name == measurement namespace
    suite_version = "echo/v1"         # provenance pin; bump when the dataset/harness changes

    def resolve(self, cfg, assets):   # offline: choose the dataset/subset
        n = int(cfg.get("n", 3))
        return DatasetHandle(version=self.suite_version, digest=f"sha256:echo-{n}", payload={"n": n})

    def prepare(self, target, dataset, driver):   # connect/provision (mock → nothing)
        return EnvHandle({"n": dataset.payload["n"]})

    def run(self, target, env, driver) -> RawArtifacts:
        return self._emit(env.payload["n"])       # real path would call the SUT here

    def mock_artifacts(self, cfg) -> RawArtifacts:  # offline fixture path (CI / mode: mock)
        return self._emit(int(cfg.get("n", 3)))

    def _emit(self, n: int) -> RawArtifacts:
        d = Path(tempfile.mkdtemp())
        rows = [{"id": f"q{i}", "correct": True} for i in range(n)]
        (d / "results.json").write_text(json.dumps(rows))
        return RawArtifacts(dir=d, manifest={"results": {"path": "results.json"}})


class OfficialEchoEvaluator(Evaluator):
    evaluator_id = "official-echo-evaluator"
    official = True                    # emits under the "echo." namespace

    def supports(self, suite_id, product) -> bool:
        return suite_id == "echo"

    def evaluate(self, raw) -> dict[str, Measurement]:
        try:
            rows = json.loads(raw.path("results").read_text())
        except Exception:              # fail-safe: never raise on a missing/corrupt artifact
            return {}
        total = len(rows) or 1
        passed = sum(1 for r in rows if r.get("correct"))
        return {"echo.pass_rate": Measurement(
            value=passed / total, unit="ratio",
            reproducibility_class="deterministic", official=True, sample_count=len(rows),
        )}
```

## Register + run

```toml theme={null}
# pyproject.toml
[project.entry-points."clousight_bench.benchmark_suites"]
echo = "my_pkg.echo_suite:EchoSuite"

[project.entry-points."clousight_bench.evaluators"]
official-echo-evaluator = "my_pkg.echo_suite:OfficialEchoEvaluator"
```

```bash theme={null}
pip install -e .          # register the entry points
csbench list             # → suite:echo shows up
csbench conformance --suite echo                         # namespace + official-flag checks
printf 'target:\n  mode: mock\n' > mock.yaml
csbench run --domain agent-runtime --benchmark echo --platform local-sim --config mock.yaml   # ✅
```

<Note>
  `conformance` enforces the contract: an `official` evaluator may only emit keys
  under `<suite_id>.`, all `official=True`. That's what makes a number trustworthy.
</Note>

## Reporting progress

A suite that runs for thirty minutes writes nothing observable until it
finishes. `driver.progress` is where it can say what it is doing, so
[`csbench serve`](/docs/viewer) can show a live view of the run.

It defaults to an inert reporter (`NULL_PROGRESS`), so **doing nothing is a
valid choice**: a suite written against plugin API 3.0, which never saw this
field, is unaffected, and a unit test needs no wiring. Every call below costs a
function call and returns on a suite that is not being watched.

Six methods (`core/progress.py::ProgressReporter`):

```python theme={null}
progress.phase(label: str, total: int = 0, *, unit: str = "", reports_progress: bool = True) -> None
progress.advance(n: int = 1, *, label: str = "") -> None
progress.step(name: str, start_ms: float, end_ms: float, *, status: str = "ok", parent: str = "") -> None
progress.sample(key: str, value: float) -> None
progress.log(msg: str, *, level: str = "INFO") -> None
progress.should_cancel() -> bool
```

`phase` starts a named unit of work (`total=0` when the size is unknown);
`advance` ticks through it; `step` records a *finished* interval so it can be
drawn on the live waterfall; `sample` publishes an in-flight observation that
nothing scored is ever derived from; `log` adds one line to the log pane
(this package's own log records are mirrored there automatically).

Step timings are milliseconds from **one origin per run**. Use
`clousight_bench.suites._progress.StepClock` so the steps emitted from
`prepare()` and the ones emitted from `run()` share a frame — and build them
from the marks a measurement already took, never from a second, disagreeing
timing.

### Never report from inside a measured window

A progress write is a file append: cheap, but not free. **If a region's own
wall clock feeds a measurement, write nothing inside it.**

Per-query reporting is usually safe, because the metric is built from intervals
measured tightly around each query and the report lands in the gap between two
of them — which nothing measures. That is exactly how the DuckDB TPC suites do
it (`suites/_duckdb_tpc.py`):

```python theme={null}
def _reported(row: dict[str, Any], start: float, end: float) -> None:
    """One finished query — drawn on the waterfall, sampled, counted.

    Runs between two queries: the row's ``latency_ms`` was already taken,
    and every measurement this path feeds (total runtime, geomean) sums
    per-query intervals, so the gap this occupies is not measured.
    """
    name = f"{self.suite_id}.q{row['query_nr']}"
    progress.step(name, clock.ms(start), clock.ms(end), parent=parent)
    progress.sample(f"{self.suite_id}.latency_ms", float(row["latency_ms"]))
    progress.advance()
    raise_if_cancelled(progress, f"{self.suite_id} query set")

progress.phase("Query set", total=len(query_ids), unit="query")
```

The TPC official-mode **Throughput test is the counter-example**: its
`elapsed_s` *is* the metric (`Throughput@Size`, the `T_TT` term). Anything
written inside that window would put the reporter's I/O into the number. So the
phase declares up front that it cannot tick, only the cancel poll runs inside,
and the per-stream, per-query steps are replayed from the intervals the phase
already measured once the window has closed
(`suites/_tpc_official/phases.py`, `_progress.TpcProgress.replay_throughput`):

```python theme={null}
progress.phase("Throughput Test", total=total_queries, unit="query", reports_progress=False)
tp_start = clock()
doc["throughput"] = run_throughput(..., poll=lambda: progress.check_cancel("throughput test"))
progress.replay_throughput(doc["throughput"], start=tp_start)
```

`reports_progress=False` means "I know my size but cannot tick through it". The
viewer then draws an indeterminate bar and says why, instead of a 0/396 bar
that reads as a hang.

### Name steps the way the sealed trajectory names them

The live waterfall and the sealed waterfall must call the same thing the same
name, or two views of one run look like two runs. If your suite writes a
`trajectory` artifact (or reconstructs spans from artifacts, as
`_tpc_official/trace.py` does), use those span names verbatim for `step()`.
`TpcProgress.name()` exists for exactly this: it builds `<suite_id>.<leaf>` so
both sides derive the name from one place.

### Cancelling: raise, never return

Poll `should_cancel()` in any loop that can run for minutes. It is a `stat`
cached for 250 ms, so a per-item poll is fine.

When it fires, **raise**. `raise_if_cancelled(progress, where)` raises
`RunCancelled`, a `KeyboardInterrupt` subclass, which the orchestrator already
handles correctly: teardown runs, an `interrupted` record is persisted, nothing
provisioned is orphaned.

<Warning>
  Never *return early* on a cancel. A truncated query set that returned normally
  would reach the evaluator looking like a finished run, and would be scored as
  one.
</Warning>

## Going further (optional)

* **Per-item scores** — implement `Evaluator.items(raw) -> list[ItemResult]` to
  emit per-example evidence; scalar measurements become their aggregation (with
  confidence intervals + per-`group` breakdowns). See [Architecture](/docs/architecture).
* **Add-on [metrics](/docs/metrics)** — bind extra metrics with
  `extra_metric_ids` (or a run's `params.extra_metrics`); they score over your
  `items` and emit `<suite_id>.<metric_id>`.
* **[Judge-based scoring](/docs/judges)** — add the `response-quality` metric (or
  your own) and config-connect a judge via `params.judge`.
* **Config-connect a real SUT** — read `target.endpoint` / `target.credentials_ref`
  in `prepare`/`run`; for LLM suites reuse `clousight_bench.suites.llm_common`
  (`resolve_endpoint`, `chat_once`, SSRF-guarded).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.