GSM8K suite (llm domain)

GSM8K (grade-school math word problems, openai/gsm8k, MIT) is the second suite on the llm domain, alongside MMLU. Same shape: run the recognized benchmark unmodified against a managed LLM endpoint and report objective accuracy (exact numeric match, deterministic) + serving dimensions (latency, token cost). It tests reasoning rather than knowledge.

# offline mock (no network, no key):
printf 'target:\n  mode: mock\n' > mock.yaml
csbench run --domain llm --benchmark gsm8k --platform llm-mock --config mock.yaml

# real run against a managed LLM (config-connect):
printf 'target:\n  mode: runtime\n  endpoint: "https://.../v1"\n  model: qwen-max\n  credentials_ref: env:DASHSCOPE_API_KEY\nparams:\n  limit: 100\n' > qwen.yaml
csbench run --domain llm --benchmark gsm8k --platform llm-endpoint --config qwen.yaml

Retries against the endpoint follow the same opt-in params.retry policy as MMLU: off by default, max_attempts / backoff_base_s (default 0.2) / backoff_max_s (default 5.0) bound it, only 429/5xx/timeouts/connection errors are retried (never other 4xx), and enabling it changes the run’s dataset digest and the dataset version label, so a retried run is never comparable with a clean one. params.retry is not target.retries, which spells the same keys but bounds the control plane only. Measurements: gsm8k.accuracy (deterministic) + gsm8k.{avg_latency_ms,total_tokens,cost_usd} (environmental), all gsm8k.-namespaced and official=True (provenance). gsm8k.avg_latency_ms is always the successful attempt alone; with params.retry enabled, gsm8k.retry_count / gsm8k.retry_overhead_ms also appear (official=False), reporting the retry cost separately. A small real sample ships as a fixture; full-dataset fetch is a future enhancement.

Get started

Core concepts

Build a benchmark

Benchmarks

Operate

Reference