> ## Documentation Index
> Fetch the complete documentation index at: https://docs.clousight.com/llms.txt
> Use this file to discover all available pages before exploring further.

# MMLU suite (llm domain)

> Benchmark a managed LLM endpoint with MMLU — objective accuracy + serving latency/cost.

The `llm` domain benchmarks the **managed large language model a cloud vendor
sells** (Bedrock, DashScope/Qwen, Vertex, or any OpenAI-compatible gateway) as a
product. Unlike `agent-runtime` (which benchmarks runtime engineering), here the
system under test is the **model endpoint** itself. **MMLU** is the first suite.

This is *not* subjective LLM-judging: we run the recognized MMLU benchmark
unmodified and report **objective multiple-choice accuracy** (deterministic) plus
the **serving dimensions** (latency, token cost) that make a managed LLM a cloud
product.

## The connection abstraction

The SUT is an OpenAI-compatible endpoint, resolved from the run `Target`:

| Platform | connects to |
| - | - |
| `llm-mock` | offline reference — replays a bundled answer fixture |
| `llm-endpoint` | an **already-running** managed LLM: `endpoint` (base URL) + `model` + `credentials_ref` |

```bash theme={null}
# offline mock (no network, no key) — the CI path:
printf 'target:\n  mode: mock\n' > mock.yaml
csbench run --domain llm --benchmark mmlu --platform llm-mock --config mock.yaml

# real run against a managed LLM (config-connect):
printf 'target:\n  mode: runtime\n  endpoint: "https://dashscope.aliyuncs.com/compatible-mode/v1"\n  model: qwen-max\n  credentials_ref: env:DASHSCOPE_API_KEY\nparams:\n  limit: 100\n' > qwen.yaml
export DASHSCOPE_API_KEY=...      # the key the credentials_ref points at
csbench run --domain llm --benchmark mmlu --platform llm-endpoint --config qwen.yaml
```

`params.subjects` filters MMLU subjects; `params.limit` caps the question count.

Retrying a failed endpoint call is opt-in and off by default: absent
`params.retry`, every question is one attempt, exactly as before this feature
existed. Set `params.retry.max_attempts` (total attempts including the first) to
retry, with `params.retry.backoff_base_s` (default `0.2`) /
`params.retry.backoff_max_s` (default `5.0`) bounding the exponential backoff
between attempts — the wait after the *n*-th failure is
`backoff_base_s × 2ⁿ⁻¹` seconds, capped at `backoff_max_s`. All three are
validated when the dataset resolves: a `max_attempts` below 1, or a negative
backoff, fails the run with a `ValueError` naming the key rather than being
silently clamped. Only a 429, a 5xx, a timeout, or a connection error is
retried — any other 4xx is the request's own fault, and retrying it would only
delay a bug you need to see, not fix it. **Enabling `params.retry` changes the
run's dataset digest**, and relabels the dataset version from
`cais-mmlu/sample-v1` to `cais-mmlu/sample-v1/retry-<max_attempts>`, the same
way a YCSB disruption plan changes what is being measured — so a retried run is
never comparable with a clean one:

```bash theme={null}
printf 'target:\n  mode: runtime\n  endpoint: "https://.../v1"\n  model: qwen-max\n  credentials_ref: env:DASHSCOPE_API_KEY\nparams:\n  retry:\n    max_attempts: 3\n    backoff_base_s: 0.5\n    backoff_max_s: 5\n' > qwen-retry.yaml
csbench run --domain llm --benchmark mmlu --platform llm-endpoint --config qwen-retry.yaml
```

`params.retry` is **not** `target.retries`. The two blocks spell the same three
key names, but `target.retries` (whose `max_attempts` defaults to `3`) bounds the
*control plane* — creating a runtime, polling its status — and has never retried
a measured `/chat/completions` call. Setting it in the hope of retrying your LLM
calls gets you no retries, no digest change and no warning: only `params.retry`
reaches the measured path.

## What it measures

The `official-mmlu-evaluator` emits `mmlu.*` measurements:

| Measurement | Meaning | class |
| - | - | - |
| `mmlu.accuracy` | correct / total (objective MMLU metric) | `deterministic` |
| `mmlu.avg_latency_ms` | mean per-question latency | `environmental` |
| `mmlu.total_tokens` | prompt + completion tokens | `environmental` |
| `mmlu.cost_usd` | tokens × seed price | `environmental` |

All are `official=True` (a provenance flag — emitted by the suite's recognized
evaluator, not an audit claim). Token/latency dimensions are omitted if the run
recorded no usage. `mmlu.avg_latency_ms` is always the successful attempt alone,
retried or not; with `params.retry` enabled, two more measurements appear —
`mmlu.retry_count` (attempts beyond the first) and `mmlu.retry_overhead_ms`
(time spent on failed attempts and backoff) — both `official=False` and
`environmental`, since they describe this harness against this endpoint on this
run, not a property of the benchmark.

## Deliberate scope (slice 1)

* A small **real MMLU sample** (cais/mmlu, MIT) ships as a fixture so the mock /
  reference paths need no network; the real path queries the configured endpoint
  over those questions. Full-dataset (all 57 subjects / \~14k questions) fetch is a
  future enhancement — today it is labeled a sample, not the full leaderboard run.
* 0-shot prompting; the answer letter is parsed with word-boundary matching.
* Cloud-provisioned model endpoints (vs bring-your-own) attach on the same seam.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.