> ## Documentation Index
> Fetch the complete documentation index at: https://docs.clousight.com/llms.txt
> Use this file to discover all available pages before exploring further.

# GSM8K suite (llm domain)

> Benchmark a managed LLM endpoint with GSM8K — objective math accuracy + serving latency/cost.

GSM8K (grade-school math word problems, openai/gsm8k, MIT) is the second suite on
the `llm` domain, alongside [MMLU](/docs/mmlu-suite). Same shape: run the
recognized benchmark unmodified against a managed LLM endpoint and report
**objective accuracy** (exact numeric match, deterministic) + serving dimensions
(latency, token cost). It tests reasoning rather than knowledge.

```bash theme={null}
# offline mock (no network, no key):
printf 'target:\n  mode: mock\n' > mock.yaml
csbench run --domain llm --benchmark gsm8k --platform llm-mock --config mock.yaml

# real run against a managed LLM (config-connect):
printf 'target:\n  mode: runtime\n  endpoint: "https://.../v1"\n  model: qwen-max\n  credentials_ref: env:DASHSCOPE_API_KEY\nparams:\n  limit: 100\n' > qwen.yaml
csbench run --domain llm --benchmark gsm8k --platform llm-endpoint --config qwen.yaml
```

Retries against the endpoint follow the same opt-in `params.retry` policy as
[MMLU](/docs/mmlu-suite): off by default, `max_attempts` / `backoff_base_s`
(default `0.2`) / `backoff_max_s` (default `5.0`) bound it, only
429/5xx/timeouts/connection errors are retried (never other 4xx), and **enabling
it changes the run's dataset digest** and the dataset version label, so a
retried run is never comparable with a clean one. `params.retry` is not
`target.retries`, which spells the same keys but bounds the control plane only.

Measurements: `gsm8k.accuracy` (deterministic) + `gsm8k.{avg_latency_ms,total_tokens,cost_usd}`
(environmental), all `gsm8k.`-namespaced and `official=True` (provenance).
`gsm8k.avg_latency_ms` is always the successful attempt alone; with
`params.retry` enabled, `gsm8k.retry_count` / `gsm8k.retry_overhead_ms` also
appear (`official=False`), reporting the retry cost separately. A small
real sample ships as a fixture; full-dataset fetch is a future enhancement.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.