> ## Documentation Index
> Fetch the complete documentation index at: https://docs.clousight.com/llms.txt
> Use this file to discover all available pages before exploring further.

# HumanEval suite (llm domain)

> Benchmark a managed LLM with HumanEval — pass@1 correctness by executing generated code against unit tests.

The `llm` domain benchmarks the **managed large language model a cloud vendor
sells** as a product. **HumanEval** (openai/openai\_humaneval, MIT) is the first
suite that scores the model by **running the code it writes**: for each problem
the SUT completes a function signature + docstring, and the completion is
executed together with the problem's self-contained `check(candidate)` unit test.
The canonical `pass@1` metric is the fraction of problems whose test passes.

This is objective, not subjective judging: a candidate either passes its unit
test or it does not (`deterministic`).

## The three paths

| Path | connects to | executes code? |
| - | - | - |
| `llm-mock` (default) | offline — replays a bundled pre-scored fixture | no |
| `llm-mock` + `params.execute=true` | offline — runs the dataset's **canonical** solutions through the sandboxed executor | yes (vetted bundled code) |
| `llm-endpoint` | an **already-running** managed LLM: `endpoint` + `model` + `credentials_ref` | yes (**model output — untrusted**) |

```bash theme={null}
# offline fixture path (no network, no key, no execution) — the CI smoke path:
printf 'target:\n  mode: mock\n' > mock.yaml
csbench run --domain llm --benchmark human-eval --platform llm-mock --config mock.yaml

# reference execution — really runs the gold solutions through the sandbox
# (no LLM); proves the execution + scoring harness works end to end:
printf 'target:\n  mode: mock\nparams:\n  execute: true\n' > exec.yaml
csbench run --domain llm --benchmark human-eval --platform llm-mock --config exec.yaml

# real run against a managed LLM (config-connect) — executes the MODEL's code.
# params.allow_code_execution: true is REQUIRED (explicit opt-in) since this runs
# untrusted model output locally; run it in an isolated environment.
printf 'target:\n  mode: runtime\n  endpoint: "https://dashscope.aliyuncs.com/compatible-mode/v1"\n  model: qwen-max\n  credentials_ref: env:DASHSCOPE_API_KEY\nparams:\n  allow_code_execution: true\n' > qwen.yaml
export DASHSCOPE_API_KEY=...
csbench run --domain llm --benchmark human-eval --platform llm-endpoint --config qwen.yaml
```

`params.limit` caps the number of problems. Model completions are unwrapped from
a single ` ```python … ``` ` fence before execution (real endpoints fence
code despite the prompt), so a fenced-but-correct answer is not mis-scored.

Retries against the endpoint follow the same opt-in `params.retry` policy as
[MMLU](/docs/mmlu-suite): off by default, `max_attempts` / `backoff_base_s`
(default `0.2`) / `backoff_max_s` (default `5.0`) bound it, only
429/5xx/timeouts/connection errors are retried (never other 4xx, since retrying
a malformed request only delays a real bug), and **enabling it changes the run's
dataset digest** and the dataset version label, so a retried run is never
comparable with a clean one. `params.retry` is not `target.retries`, which
spells the same keys but bounds the control plane only. Retries apply to the
*generation* call; the sandboxed execution below is never retried.

## Sandbox & safety

Execution reuses the first-party `core.sandbox` helper: each candidate runs in a
**fresh subprocess** with a **secret-stripped environment** (so model code cannot
read the operator's `*_API_KEY`s out of `os.environ`), a wall-clock `timeout`, a
new session so a timeout kills the whole process group, and POSIX `rlimit` caps
(CPU, address space, file size, fd count) applied in a `preexec_fn`, in a
throwaway working directory. The `llm-endpoint` path additionally **requires an
explicit `params.allow_code_execution: true`** opt-in and SSRF-guards the endpoint
URL (http(s) only, no cloud-metadata/link-local host) before sending the Bearer
key. As `core.sandbox` documents, this is defence-in-depth, **not a
container** — it bounds runaway CPU/memory/output, not filesystem or network
access. The offline reference path executes only the dataset's **vetted canonical
code**; executing **model-generated** completions (`llm-endpoint`) is untrusted
and is your decision to make in your own isolated environment.

## What it measures

The `official-humaneval-evaluator` emits `human-eval.*` measurements:

| Measurement | Meaning | class |
| - | - | - |
| `human-eval.pass_at_1` | problems\_passed / total (canonical HumanEval metric) | `deterministic` |
| `human-eval.avg_latency_ms` | mean per-problem **sandbox execution** wall time — not a generation latency | `environmental` |
| `human-eval.total_tokens` | prompt + completion tokens | `environmental` |
| `human-eval.cost_usd` | tokens × seed price | `environmental` |

All are `official=True` (a provenance flag — emitted by the suite's recognized
evaluator, not an audit claim). Token/latency dimensions are omitted when the run
recorded no usage (e.g. the reference/fixture path).

**`human-eval.avg_latency_ms` is not an LLM latency.** Unlike
`mmlu.avg_latency_ms` / `gsm8k.avg_latency_ms`, which are the generation call's
own duration, this one is the mean wall time of running each generated solution
in the sandbox — a different loop, containing no endpoint time at all. Retries
therefore cannot move it in either direction, and comparing it against another
suite's `avg_latency_ms` compares two unrelated quantities. What the retries of
the *generation* calls cost is reported separately: with `params.retry` enabled,
`human-eval.retry_count` / `human-eval.retry_overhead_ms` also appear
(`official=False`).

## Deliberate scope (slice 1)

* A small **real HumanEval sample** (openai/openai\_humaneval, MIT) ships as a
  fixture pinned to the dataset main-commit, so the mock / reference paths need no
  network. Full-164-problem fetch is a future enhancement — today it is labeled a
  sample, not the full leaderboard run.
* Greedy single-sample `pass@1`; `pass@k` sampling is a future enhancement.
* This suite is the code-execution substrate other execution-scored benchmarks
  (SciCode, USACO) build on.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.