MMLU suite (llm domain)

The llm domain benchmarks the managed large language model a cloud vendor sells (Bedrock, DashScope/Qwen, Vertex, or any OpenAI-compatible gateway) as a product. Unlike agent-runtime (which benchmarks runtime engineering), here the system under test is the model endpoint itself. MMLU is the first suite. This is not subjective LLM-judging: we run the recognized MMLU benchmark unmodified and report objective multiple-choice accuracy (deterministic) plus the serving dimensions (latency, token cost) that make a managed LLM a cloud product.

The connection abstraction

The SUT is an OpenAI-compatible endpoint, resolved from the run Target:

Platform	connects to
`llm-mock`	offline reference — replays a bundled answer fixture
`llm-endpoint`	an already-running managed LLM: `endpoint` (base URL) + `model` + `credentials_ref`

# offline mock (no network, no key) — the CI path:
printf 'target:\n  mode: mock\n' > mock.yaml
csbench run --domain llm --benchmark mmlu --platform llm-mock --config mock.yaml

# real run against a managed LLM (config-connect):
printf 'target:\n  mode: runtime\n  endpoint: "https://dashscope.aliyuncs.com/compatible-mode/v1"\n  model: qwen-max\n  credentials_ref: env:DASHSCOPE_API_KEY\nparams:\n  limit: 100\n' > qwen.yaml
export DASHSCOPE_API_KEY=...      # the key the credentials_ref points at
csbench run --domain llm --benchmark mmlu --platform llm-endpoint --config qwen.yaml

params.subjects filters MMLU subjects; params.limit caps the question count. Retrying a failed endpoint call is opt-in and off by default: absent params.retry, every question is one attempt, exactly as before this feature existed. Set params.retry.max_attempts (total attempts including the first) to retry, with params.retry.backoff_base_s (default 0.2) / params.retry.backoff_max_s (default 5.0) bounding the exponential backoff between attempts — the wait after the n-th failure is backoff_base_s × 2ⁿ⁻¹ seconds, capped at backoff_max_s. All three are validated when the dataset resolves: a max_attempts below 1, or a negative backoff, fails the run with a ValueError naming the key rather than being silently clamped. Only a 429, a 5xx, a timeout, or a connection error is retried — any other 4xx is the request’s own fault, and retrying it would only delay a bug you need to see, not fix it. Enabling params.retry changes the run’s dataset digest, and relabels the dataset version from cais-mmlu/sample-v1 to cais-mmlu/sample-v1/retry-<max_attempts>, the same way a YCSB disruption plan changes what is being measured — so a retried run is never comparable with a clean one:

printf 'target:\n  mode: runtime\n  endpoint: "https://.../v1"\n  model: qwen-max\n  credentials_ref: env:DASHSCOPE_API_KEY\nparams:\n  retry:\n    max_attempts: 3\n    backoff_base_s: 0.5\n    backoff_max_s: 5\n' > qwen-retry.yaml
csbench run --domain llm --benchmark mmlu --platform llm-endpoint --config qwen-retry.yaml

params.retry is not target.retries. The two blocks spell the same three key names, but target.retries (whose max_attempts defaults to 3) bounds the control plane — creating a runtime, polling its status — and has never retried a measured /chat/completions call. Setting it in the hope of retrying your LLM calls gets you no retries, no digest change and no warning: only params.retry reaches the measured path.

What it measures

The official-mmlu-evaluator emits mmlu.* measurements:

Measurement	Meaning	class
`mmlu.accuracy`	correct / total (objective MMLU metric)	`deterministic`
`mmlu.avg_latency_ms`	mean per-question latency	`environmental`
`mmlu.total_tokens`	prompt + completion tokens	`environmental`
`mmlu.cost_usd`	tokens × seed price	`environmental`

All are official=True (a provenance flag — emitted by the suite’s recognized evaluator, not an audit claim). Token/latency dimensions are omitted if the run recorded no usage. mmlu.avg_latency_ms is always the successful attempt alone, retried or not; with params.retry enabled, two more measurements appear — mmlu.retry_count (attempts beyond the first) and mmlu.retry_overhead_ms (time spent on failed attempts and backoff) — both official=False and environmental, since they describe this harness against this endpoint on this run, not a property of the benchmark.

Deliberate scope (slice 1)

A small real MMLU sample (cais/mmlu, MIT) ships as a fixture so the mock / reference paths need no network; the real path queries the configured endpoint over those questions. Full-dataset (all 57 subjects / ~14k questions) fetch is a future enhancement — today it is labeled a sample, not the full leaderboard run.
0-shot prompting; the answer letter is parsed with word-boundary matching.
Cloud-provisioned model endpoints (vs bring-your-own) attach on the same seam.

Get started

Core concepts

Build a benchmark

Benchmarks

Operate

Reference

The connection abstraction

What it measures

Deliberate scope (slice 1)

​The connection abstraction

​What it measures

​Deliberate scope (slice 1)

The connection abstraction

What it measures

Deliberate scope (slice 1)