HumanEval suite (llm domain)

The llm domain benchmarks the managed large language model a cloud vendor sells as a product. HumanEval (openai/openai_humaneval, MIT) is the first suite that scores the model by running the code it writes: for each problem the SUT completes a function signature + docstring, and the completion is executed together with the problem’s self-contained check(candidate) unit test. The canonical pass@1 metric is the fraction of problems whose test passes. This is objective, not subjective judging: a candidate either passes its unit test or it does not (deterministic).

The three paths

Path	connects to	executes code?
`llm-mock` (default)	offline — replays a bundled pre-scored fixture	no
`llm-mock` + `params.execute=true`	offline — runs the dataset’s canonical solutions through the sandboxed executor	yes (vetted bundled code)
`llm-endpoint`	an already-running managed LLM: `endpoint` + `model` + `credentials_ref`	yes (model output — untrusted)

# offline fixture path (no network, no key, no execution) — the CI smoke path:
printf 'target:\n  mode: mock\n' > mock.yaml
csbench run --domain llm --benchmark human-eval --platform llm-mock --config mock.yaml

# reference execution — really runs the gold solutions through the sandbox
# (no LLM); proves the execution + scoring harness works end to end:
printf 'target:\n  mode: mock\nparams:\n  execute: true\n' > exec.yaml
csbench run --domain llm --benchmark human-eval --platform llm-mock --config exec.yaml

# real run against a managed LLM (config-connect) — executes the MODEL's code.
# params.allow_code_execution: true is REQUIRED (explicit opt-in) since this runs
# untrusted model output locally; run it in an isolated environment.
printf 'target:\n  mode: runtime\n  endpoint: "https://dashscope.aliyuncs.com/compatible-mode/v1"\n  model: qwen-max\n  credentials_ref: env:DASHSCOPE_API_KEY\nparams:\n  allow_code_execution: true\n' > qwen.yaml
export DASHSCOPE_API_KEY=...
csbench run --domain llm --benchmark human-eval --platform llm-endpoint --config qwen.yaml

params.limit caps the number of problems. Model completions are unwrapped from a single ```python … ``` fence before execution (real endpoints fence code despite the prompt), so a fenced-but-correct answer is not mis-scored. Retries against the endpoint follow the same opt-in params.retry policy as MMLU: off by default, max_attempts / backoff_base_s (default 0.2) / backoff_max_s (default 5.0) bound it, only 429/5xx/timeouts/connection errors are retried (never other 4xx, since retrying a malformed request only delays a real bug), and enabling it changes the run’s dataset digest and the dataset version label, so a retried run is never comparable with a clean one. params.retry is not target.retries, which spells the same keys but bounds the control plane only. Retries apply to the generation call; the sandboxed execution below is never retried.

Sandbox & safety

Execution reuses the first-party core.sandbox helper: each candidate runs in a fresh subprocess with a secret-stripped environment (so model code cannot read the operator’s *_API_KEYs out of os.environ), a wall-clock timeout, a new session so a timeout kills the whole process group, and POSIX rlimit caps (CPU, address space, file size, fd count) applied in a preexec_fn, in a throwaway working directory. The llm-endpoint path additionally requires an explicit params.allow_code_execution: true opt-in and SSRF-guards the endpoint URL (http(s) only, no cloud-metadata/link-local host) before sending the Bearer key. As core.sandbox documents, this is defence-in-depth, not a container — it bounds runaway CPU/memory/output, not filesystem or network access. The offline reference path executes only the dataset’s vetted canonical code; executing model-generated completions (llm-endpoint) is untrusted and is your decision to make in your own isolated environment.

What it measures

The official-humaneval-evaluator emits human-eval.* measurements:

Measurement	Meaning	class
`human-eval.pass_at_1`	problems_passed / total (canonical HumanEval metric)	`deterministic`
`human-eval.avg_latency_ms`	mean per-problem sandbox execution wall time — not a generation latency	`environmental`
`human-eval.total_tokens`	prompt + completion tokens	`environmental`
`human-eval.cost_usd`	tokens × seed price	`environmental`

All are official=True (a provenance flag — emitted by the suite’s recognized evaluator, not an audit claim). Token/latency dimensions are omitted when the run recorded no usage (e.g. the reference/fixture path). human-eval.avg_latency_ms is not an LLM latency. Unlike mmlu.avg_latency_ms / gsm8k.avg_latency_ms, which are the generation call’s own duration, this one is the mean wall time of running each generated solution in the sandbox — a different loop, containing no endpoint time at all. Retries therefore cannot move it in either direction, and comparing it against another suite’s avg_latency_ms compares two unrelated quantities. What the retries of the generation calls cost is reported separately: with params.retry enabled, human-eval.retry_count / human-eval.retry_overhead_ms also appear (official=False).

Deliberate scope (slice 1)

A small real HumanEval sample (openai/openai_humaneval, MIT) ships as a fixture pinned to the dataset main-commit, so the mock / reference paths need no network. Full-164-problem fetch is a future enhancement — today it is labeled a sample, not the full leaderboard run.
Greedy single-sample pass@1; pass@k sampling is a future enhancement.
This suite is the code-execution substrate other execution-scored benchmarks (SciCode, USACO) build on.

Get started

Core concepts

Build a benchmark

Benchmarks

Operate

Reference

HumanEval suite (llm domain)

The three paths

Sandbox & safety

What it measures

Deliberate scope (slice 1)

​The three paths

​Sandbox & safety

​What it measures

​Deliberate scope (slice 1)

The three paths

Sandbox & safety

What it measures

Deliberate scope (slice 1)