Skip to main content
csbench serve starts a small local web server on top of a results directory and serves an embedded single-page viewer. It shows you a run while it is happening, helps you read the runs that have already finished, and — only when you ask for it — lets you edit a target config and start a run. Read-only is the default, and that is the design. Without --allow-write every write route answers 405 and names the flag, so “not allowed here” never reads as “broken”. The one exception predates the flag: a zero-byte cancel marker for a run that is still going, see Stopping a run. Sealed records are never writable, flag or no flag. Every record carries a record_digest that covers its measurements; an edit route would make “this result can be re-checked” false, and that is the promise the whole tool rests on. See Writing.

​
Usage

The printed line names the posture — (results: results, read-only) or read+write — because that is the server’s whole security argument and someone reading their scrollback should be able to tell which one is up. The command prints the actual bound address (useful with --port 0) and runs until Ctrl-C. The header carries the five sections — Suites, Platforms, Runs, Observe, Config — plus a language switcher (EN | 中文; the choice persists in localStorage and defaults to the browser language), an engineer-view toggle and a theme toggle (follows prefers-color-scheme until toggled, then persists). The Live tab carries a count badge only while something is actually running; there is no permanent “0”.

​
Watching a run happen

Start a run in one terminal and csbench serve in another. The run appears on its own — the viewer polls /api/progress every few seconds, and a strip appears above the board while anything is in flight.
  • #/live — everything in flight at once. With exactly one run it forwards straight to that run’s page.
  • #/live/:run_id — one run, fed by a single SSE connection to /api/progress/<run_id>/stream.
The run page is five panes:
  • Header — benchmark, platform, status, an elapsed clock that ticks once a second while the run is running, and the Stop run button.
  • Progress — the current stage in plain language (from the same stage glossary the record page uses), and the current phase as 12 / 99 query with a bar. An estimate of the time left is labelled est. and is withheld until at least three units have completed (web/src/lib/eta.ts): a confidently wrong “3 seconds left” costs more than a blank.
  • Execution, as it happens — a waterfall of the run so far, fed by step events instead of spans. (It is its own renderer: the sealed trace page moved to the zoomable tree described under Reading results and no longer draws this waterfall.) A live step names its parent (a suite knows what it called the enclosing phase, not what span id the tracer will mint later), so nesting is resolved by name.
  • Numbers so far — computed in the browser from the steps that have finished: how many steps are done, the median and p99 step, failed steps, the slowest step so far.
  • Log — the tail of this run’s clousight_bench log records, filterable by level. Auto-scroll is sticky-to-bottom: scroll up and it stops yanking you back.

​
”Preliminary” means it is not a measurement

Every tile in Numbers so far carries a preliminary chip, and the pane says outright where the numbers came from. That is a contract, not modesty:
  • a measurement is produced by an Evaluator, offline, over the sealed evidence, after the run has ended — it is what lands in the record and what record_digest covers;
  • the live tiles are a lagging browser-side arithmetic over whatever events have arrived so far, from a stream that is explicitly allowed to drop events.
Nothing scored is ever derived from a sample event. A screenshot of the live page is a screenshot of a partial run, and the UI is built so a reader cannot mistake it for a result.

​
Why a measured window shows an indeterminate bar

A phase can declare that it knows its size but cannot tick through it: progress.phase("Throughput Test", total=396, unit="query", reports_progress=False). The viewer then draws an indeterminate bar and says “no progress reported from inside a measured window”, instead of a 0% bar that reads as a hang. The reason is in the suite, not the UI. In the TPC official mode the Throughput test’s own wall clock is the metric (Throughput@Size / the T_TT term), so writing progress files inside that window would put the viewer’s I/O into the number. The suite therefore reports nothing from inside it and replays the per-stream, per-query steps from the intervals it already measured once the window has closed — same numbers, same layout, a few seconds late. See reporting progress for the rule a suite author has to follow.

​
Stopping a run

The Stop button is the viewer’s only mutating control, and it is deliberately two-step: arm, then confirm. Pressing it POSTs to /api/progress/<run_id>/cancel, which creates a zero-byte cancel file in the run’s progress directory. Nothing else happens over HTTP. The run picks it up on its own schedule — the orchestrator checks at every stage boundary, and a wired suite polls should_cancel() inside its loops (a cached stat, at most four per second). When it fires, the run raises RunCancelled, which is a KeyboardInterrupt subclass, so the cancel takes the path a Ctrl-C already took:
  • teardown still runs — nothing provisioned is orphaned or left billing;
  • an interrupted record is persisted, with whatever stages completed;
  • the live page swaps to a handoff card pointing at that record.
Once a cancel is requested the button stays disabled and reads “stopping — teardown is running”. The run decides when it can stop; pretending the browser is in control would be a lie.

​
When the run ends

The page does not go blank or 404. The progress directory survives for TERMINAL_GRACE_S (120 s by default) precisely so a subscriber can still read the terminal done event and the sealed record’s path out of it, and the page swaps to a handoff card linking to #/record/<run_id>. If some events were dropped to stay inside the stream caps, the page says so and says which kinds — and says that the sealed record is unaffected.

​
Reading results

Four pages, in the order a question narrows: A run is one object with four faces, selected by ?tab=: overview, per-item detail (absent on suites that emit no items), the execution trace, and the frozen configuration snapshot. Its identity bar answers “what am I looking at” before any tab is pressed, and carries Create like this, which seeds a new run from this one. Every older spelling still resolves and rewrites itself in the address bar: #/, #/record/:id[/trace|/items], #/suite/:d/:s, #/live[/:id].
  • Board groups every record by domain and then by benchmark, showing each benchmark’s platform count, run count and its newest run’s headline metrics. Domains and suites are ordered by most recent activity, and a domain card shows five benchmarks before offering to expand: a results tree that has accumulated a long tail of retired benchmarks should not push what you ran today off the first screen. A record that does not name both a domain and a suite (a schema-0.1-era run summarises to empty strings) is not folded in — a nameless, valueless card is worse than an absence — it stays visible under #/runs.
  • Suite comparison leads with the table a selection decision is made on: the latest run on each platform, one column per metric, ordered by the glossary (headline metrics first) rather than alphabetically. Only metrics some platform’s latest run actually produced get a column, so a TPC-H reference run and an official run do not produce a table of em-dashes. Trend charts for up to four metrics follow, over a bounded history (the last 30 runs per platform).
  • Record reads top-down: what it found, whether the machinery behaved, what it measured, whether the numbers can be trusted. The eleven recorded lifecycle stages collapse into one health line — “Everything behaved · 11 stages, all passed” — which expands into the full four-phase stage strip; when something failed, the line names the stage instead.
  • Trace is one zoomable tree of the run’s spans. Drag the overview strip to narrow the time window and every bar re-draws against it — the axis moves, rather than the spans outside it merely dimming. Selecting a row docks its detail on the right without pushing any row out of view, children that ran concurrently pack into lanes you can drill into, and each row prints two durations: the part inside the window, and the span’s unmoving total. The aggregate and table tabs follow underneath, over the spans the window contains. Every run has one — see the execution trace.
#/runs is the old flat table of every run, with a client-side filter box. It is genuinely useful once you already know which run you are hunting for, which is why it is still there and no longer the front door.

​
Writing

Started with --allow-write, the console can do two things. Both are fenced the same way: the shared Host guard, a required X-Csbench-Write: 1 header (a custom header is unreachable from a cross-origin <form>, which closes simple-form CSRF), and a 64 KiB cap read off Content-Length before a byte of the body is read. Targets (configs/*.yaml) can be created, edited and deleted. A name off the wire is a name, never a path: it must match ^[A-Za-z0-9._-]{1,64}$ (with . and .. excluded, since both match that pattern and name a directory) and the resolved path must still be inside the configs directory — checked after resolve(), because a symlink passes any name check. YAML is parsed before it is written, so a file that will not load never reaches the disk. Values whose key looks like a credential (secret, token, password, key, credential) come back as ***, and when that happens the file’s raw text is withheld entirely — raw text is the one thing that cannot be redacted safely, so such a target is shown but not editable here. A PUT carrying *** is refused: that body is a redacted read on its way back, and saving it would overwrite a real credential with three asterisks. Runs can be started. The request body is a fixed set of fields — domain, benchmark, platform, target name, scalar params, repeat, warmup — validated against the live plugin registry. There is no field for a command line and none for a path. repeat is capped (20) because it is what turns one click into an afternoon of billable work, and a fifth concurrent run is refused with 429 rather than queued. The run is a detached subprocess (sys.executable -m clousight_bench run --run-id ...), so the reply can name a run before it has started and the browser can open the live view on it immediately. What the form deliberately does not enforce is that a benchmark “belongs” on a platform: nothing in the registry pairs them, so any rule would be invented. It shows where each benchmark has actually run, warns before a platform that bills or an adapter that is still a skeleton, and lets you decide. The console keeps its own notes under results/.launches/ — what it was asked to start, beside the results rather than in them. That is what lets a target say how many runs used it; a sealed record does not carry the name of the config file it was handed.

​
Plain language

The viewer renders internal vocabulary through one de-jargon layer, web/src/lib/glossary.ts, under two rules. Two layers, never one. The human label leads and the raw key stays visible underneath in mono — Typical query time over tpc-h.geomean_latency_ms, Seal evidence over SEAL. Nothing becomes unauditable, and an engineer can still match a field name at a glance. This is not affected by any toggle. Never invent. An unrecognised measurement key still renders: it gets a prettified label and a format inferred from its declared unit or, cautiously, from its suffix. It never gets a direction. betterIs stays "none" unless a human wrote the spec, because calling a rising number “better” when it is actually a regression is worse than saying nothing — so an unknown metric is simply never bolded as a leader in the comparison table. The same layer explains the two things that reliably confuse readers: invalid (we never measured anything — not a failure of the system under test) versus failed (we measured and it broke), and PUBLISH: skipped, which is a durable denial that any publisher touched the sealed record, not an error. Engineer view is a persisted toggle in the header. Off, the record page shows conclusions. On, it adds a panel with the record’s fingerprints, identity, environment and extensions in full — the fields that make a run reproducible and comparable, and that the previous viewer never rendered at all. What the toggle does not hide either way is the raw key under every label.

​
The progress plane on disk

Live progress is a second, deliberately separate plane from the sealed record (src/clousight_bench/core/progress.py):
Properties that are load-bearing:
  • Disposable, and outside the record. Nothing here is covered by record_digest, nothing here can move a verdict, and the whole directory is swept once the sealed record exists. Deleting results/.progress/ at any time loses nothing but a live view.
  • Dot-prefixed on purpose. A state.json with a status field sits one careless rglob("*.json") away from being read as a result record.
  • Best-effort. Every write is contained: an IO failure logs at debug level, disables the plane for the rest of the run, and the benchmark carries on.
  • Bounded. Past the caps, the writer drops rather than grows, and it only ever drops the droppable kinds — log and sample. stage and step events are never dropped; they are the shape of the run. Dropped counts are published in state.json and surfaced in the UI.
  • Reaped. A stale directory is collected when a run starts and when the viewer starts, so a machine that lost power mid-benchmark never shows a phantom “running” row. A snapshot that still claims to be running while its pid is gone is reported as abandoned, not as running forever.
  • Versioned. state.json carries "schema": "progress/1"; a reader that does not recognise the schema ignores the directory rather than guessing.
Three environment variables tune it:

​
API

Anything else is a 404 with a JSON body — the app is a hash router, so there is no SPA fallback that would turn a typo into an HTTP 200. The SSE stream sends state frames when the snapshot changes, events frames carrying stream events, a : heartbeat comment every 15 s so an idle proxy does not drop the connection, and a final done frame. It polls the two files every 400 ms rather than watching them — the plane is deliberately plain files with no notification channel. Each stream owns a server thread for its whole life, so there is a hard cap of 8 concurrent streams (a 9th gets a 503) and a 6 hour lifetime cap per stream. Every event carries a seq, which the browser uses as a resume token after a reconnect.

​
Development and build

The viewer is a React + TypeScript + Vite + Tailwind app under web/. Its production build is committed at src/clousight_bench/resources/viewer/dist/ and shipped inside the wheel, so installing the package never requires Node.
After changing anything under web/, rebuild and commit the regenerated dist/ in the same change — tests validate the committed artifact, not the sources alone. CI’s viewer-dist job runs npm test, rebuilds, and fails if the committed dist/ is not byte-identical to the rebuild.

​
Scope and security

  • Binds 127.0.0.1 by default. It has no authentication — do not expose it on a public interface.
  • Requests whose Host header is not the bound host or a localhost alias get a 403 (DNS-rebinding guard). The guard sits above the method split, so POST can never end up behind a differently-worded copy of it.
  • Write routes are off unless --allow-write is passed, and answer 405 naming the flag when they are. Reading targets is not gated: a Targets page that could list nothing on a read-only server would be a page about a flag. Sealed records have no write route at all, in either mode.
  • The cancel endpoint predates the flag and is fenced independently: the shared Host guard, a required X-Csbench-Progress: 1 header (a custom header is unreachable from a cross-origin <form>, which closes simple-form CSRF), a refusal of any request body, and a run_id that must be a plain [A-Za-z0-9._-]+ token before it can name anything on disk. All it can ever do is create a zero-byte marker for a run that is still running.
  • Cache policy: no-store for the document and every JSON API response — it is all live state; the SSE stream sends no-cache — and public, max-age=31536000, immutable for /assets/*, whose filenames are content-hashed by the Vite build. Without the first half, upgrading csbench and reloading would serve a cached index.html naming an asset hash the new wheel does not contain: a white page with nothing in the console to explain it.
  • The built app is offline-first: everything (React, ECharts, styles, both locale dictionaries) is bundled into the committed dist/, system font stack only, and no external requests — a build-time guard plus a test enforce that no dist file contains an absolute URL.
  • It is served under a strict CSP (default-src 'none'; script-src 'self'; style-src 'self'; connect-src 'self') with nothing inline (tests enforce that index.html has no inline scripts, styles, or handlers), and server-provided strings are always rendered as text, never as markup (dangerouslySetInnerHTML is banned by test).
  • Static serving is containment-checked inside the packaged dist/ directory before any path joining; .., separators, colons and unknown extensions return 404.
  • Artifact and trace reads are resolved and must stay inside the results directory. /api/meta publishes the results directory’s basename only, never its full path — and the record path in a progress handoff is relative for the same reason.

​
The execution trace

A run’s trace is one tree: csbench.run at the root, its lifecycle stages beneath, and — for a benchmark that reports them — that benchmark’s own spans nested inside the stage that produced them, down to a single query. That took two things, and neither was obvious from the outside. The stages needed a real timeline. emit_run_trace used to lay them end-to-end from their durations, so only how long a stage took was real and when it happened was invented. run.stage_spans now records each timed stage’s real [start_ms, end_ms] from the run’s start. Two consequences: a suite’s spans, which carry their own timestamps, land where they belong; and a gap between two stages is visible, which a list of durations cannot show — the first run with real marks revealed five seconds before PREFLIGHT that nobody could previously see. The field is optional and additive, and a record written without it still renders through the old reconstruction. The two halves had to be joined. They were already the same trace — a suite receives the run’s trace_id on its DriverContext — but the suite’s root span had no parent and lived in its own file, so the viewer showed one or the other. emit_run_trace now re-emits the suite’s trajectory as children of csbench.stage.EXECUTE. The suite’s own artifact is left byte-identical: it is the SUT’s unmodified account and its sha256 is pinned in the record, so the run trace becomes the merged view while the artifact stays the primary source. A span whose timestamps fall outside the stage window keeps them — clamping would hide exactly the disagreement worth seeing.

​
What a reader gets

/api/record/<run_id>/trajectory says in its source field which of three things it returned, decided by looking at the spans rather than at a version:
  • full — results/traces/<trace_id>.jsonl with the benchmark’s spans merged in. The whole chain. This is the normal case.
  • artifact — the record’s kind=trajectory sidecar, for a record written before the merge existed. Its detail is there; the stages are not.
  • lifecycle — a run trace with no benchmark spans in it, because the benchmark reported none. Every run has at least this.
A declared but unreadable artifact is still a 404 when the run trace cannot supersede it, which is exactly the pre-merge case: the record claims that file, and rendering six stage bars instead would let a reader mistake the frame for the whole run.

​
Which benchmarks report

A benchmark gets spans inside EXECUTE by writing a kind=trajectory artifact. The bundled ones do, on their real paths: Mock paths seal nothing. The bundled fixtures carry canned latencies, and a waterfall of invented numbers is worse than no waterfall, so a mode: mock run shows lifecycle and says so. See reporting progress for writing one. The same rule governs both: a span may be recorded from marks a measured region already took — the per-query spans above are literally the intervals latency_ms was computed from — but nothing may be written inside a window whose own wall clock is the measurement. MMLU and GSM8K are the one documented exception to the first half of that rule. Their live item step is drawn from the item loop’s own marks, around the whole call, while the sealed latency_ms is a separate reading chat_once takes from just before the HTTP request to just after the response is parsed. The two differ on every run by the per-call setup between them, and under an enabled params.retry by the failed attempts and backoff sleeps as well — so the live number is always the larger of the two. That is deliberate: the live sample answers “how long is this item taking”, the sealed one answers “how fast is the endpoint”.

​
Span shape

The primary span shape is the v3 OTel-native span (core/sut_span.py): span_id / trace_id / parent_span_id / name / start_unix_nano / end_unix_nano / status / attributes, where status is OTel OK / ERROR / UNSET and attributes must contain at least one semconv-namespaced key as the kind discriminator — gen_ai.* (model and tool calls), db.* (queries) or csbench.* (phases, streams, and the run’s own lifecycle). Legacy v2 spans (parent_id, t_start / t_end, attrs, a literal kind) are still accepted and normalized by the viewer. The viewer colours by that kind, and treats the run’s own lifecycle as chrome rather than as a series: it takes the recessive axis colour, so the frame does not compete with the work hanging inside it. See the result-record schema (src/clousight_bench/resources/schemas/result-record-0.4.schema.json) for how artifacts are referenced from a record.