> ## Documentation Index
> Fetch the complete documentation index at: https://docs.clousight.com/llms.txt
> Use this file to discover all available pages before exploring further.

# SWE-bench live smoke runbook

> Run the 3-instance SWE-bench Verified smoke on real Aliyun infrastructure — submit, verify, fetch, teardown.

This runbook takes the committed plan `configs/swe-bench-smoke.plan.yaml` — 3 real
SWE-bench Verified instances in **oracle** mode (gold-patch replay, zero LLM spend) —
through a full real-cloud pass: the AgentRun-hosted agent produces the predictions,
the upstream `swebench` harness evaluates them in Docker on an ECS driver host, and
the results land back on your laptop via OSS.

The plan's *shape* is already proven offline: `tests/test_swe_submit_e2e.py` submits
the exact committed file through the real `csbench submit` code path and executes its
task entry end-to-end in mock mode. What only a live run can prove is the wiring —
Docker, the dataset download, the AgentRun invocation, billing — hence this runbook.

## Preconditions

* **Aliyun balance ≥ ¥100.** Postpaid ECS creation is rejected below this account
  balance — the submit will fail at `terraform apply` with an opaque quota-like error.
* **RAM credentials.** MAIN-account AK exported for terraform
  (`ALICLOUD_ACCESS_KEY` / `ALICLOUD_SECRET_KEY` / `ALICLOUD_REGION`); the controller's
  own SDK calls use the sub-account instance role that `controller.tf` provisions.
* **`configs/agent-runtime-aliyun.local.yaml`** (gitignored, never committed) with
  `target.region`, `target.blob_bucket`, the credential *env-var names* under
  `target.auth_env`, and `target.arms_license_key` for telemetry.
* **`target.mode: real` in that local config.** Without it the suite bridge defaults
  to mock — you would pay for a full-price ECS campaign that replays canned fixtures
  (`csbench submit` prints a loud warning if a `suite:` task is submitted non-real).
* **ARMS license key** present in that local config (trace/metric export).
* **`llm` mode only — key delivery is NOT yet wired for prod submit.**
  `DASHSCOPE_API_KEY` must be present in the **DRIVER host's environment at
  provision time**: the SUT client forwards it into the CreateAgentRuntime
  provision env, but prod cloud-init does **not** deliver it to the driver today.
  Two working options: (i) submit with `controller_debug=true`, SSH to the
  controller and `export DASHSCOPE_API_KEY=...` into the orchestrator's
  environment *before* the campaign claims the suite task; or (ii) run llm mode
  from a locally-driven (non-submit) execution where your own shell environment
  carries the key. Oracle mode needs no LLM key at all.

## The plan

`configs/swe-bench-smoke.plan.yaml` (committed; placeholders only — real values live
in the gitignored local config):

```yaml theme={null}
version: "1"
domain: agent-runtime
platform: aliyun-agentrun

# Campaign-wide hard cap in USD; the controller stops before provisioning a
# billable task that would cross it.
cost_budget: 10.0

# The controller ECS doubles as the docker-capable SWE-bench driver host.
# Every key maps to a `-var controller_*` terraform var; unknown keys are
# rejected at submit time.
driver:
  install_docker: true          # the upstream harness runs docker on the driver
  system_disk_size: 120         # GiB — 3 instance images + base easily exceed the 40 default
  # NO mirror address needed: the driver auto-detects its image strategy at boot
  # (probes Docker Hub; direct-pull when reachable, ACR when blocked, fail-loud
  # otherwise — region-agnostic, see driver_image.py). Uncomment only to FORCE a
  # specific mirror:
  #   docker_registry_mirror: "https://<your-own-mirror>"
  # huggingface.co is unreliable from cn regions — pin the mirror for the
  # SWE-bench_Verified dataset auto-download.
  hf_endpoint: "https://hf-mirror.com"
  # image builds need ≥4c8g — the default e-c1m2.large is too small for the harness.
  instance_type: "ecs.c6.xlarge"

tasks:
  - task_id: "suite:swe-bench"
    params:
      # oracle = gold-patch replay through the REAL AgentRun SUT + REAL harness:
      # proves the whole pipe with zero LLM spend. Expected: resolved == 1.0.
      # Switch to `llm` for the paid Qwen variant (needs DASHSCOPE_API_KEY on
      # the driver; yields a real swe-bench.cost_per_resolved).
      agent_kind: oracle
      # The 3 bundled real SWE-bench Verified rows
      # (src/clousight_bench/suites/swe_bench/fixtures/instances_subset.json).
      instance_ids:
        - django__django-11099
        - sympy__sympy-20590
        - pytest-dev__pytest-7205
```

You configure **no mirror address**. The driver host probes Docker Hub at boot and
picks its image strategy itself (`driver_image.py`): direct pull when reachable
(overseas / unblocked regions), the account's own ACR endpoint when Docker Hub is
blocked, or a loud abort otherwise — the decision is logged to the campaign. Set
`driver.docker_registry_mirror` only to FORCE a specific mirror. Unknown `driver:`
keys are rejected at submit time.

## Command sequence

```bash theme={null}
# 1. Submit — writes the launch spec to OSS, terraform-applies controller + NAT.
csbench submit configs/swe-bench-smoke.plan.yaml \
  --config configs/agent-runtime-aliyun.local.yaml \
  --watchdog-timeout 5400
# prints the campaign_id, e.g. camp-ab12cd34

# 2. The laptop may go offline. Poll progress (reads OSS only):
csbench status  camp-ab12cd34 --config configs/agent-runtime-aliyun.local.yaml
csbench logs    camp-ab12cd34 --config configs/agent-runtime-aliyun.local.yaml

# 3. When status shows done, pull the records:
csbench fetch   camp-ab12cd34 --config configs/agent-runtime-aliyun.local.yaml \
  --dest results/swe-smoke

# 4. Backstop cleanup — ALWAYS run if self-destruct is not confirmed:
csbench teardown camp-ab12cd34 --config configs/agent-runtime-aliyun.local.yaml
```

## Live-verification checklist (8 points)

Work through these on the first live pass — each one covers a seam that mock-mode
tests cannot reach:

1. **Harness report filename + keys.** With installed `swebench>=3.0` the report is
   written to `<report_dir>/<model_name>.<run_id>.json` (slashes in the model
   name become `__`) with top-level keys
   `resolved_instances` / `total_instances` / `resolved_ids`. The suite fails loudly
   on schema drift — a clean pass here pins the version as compatible.
2. **Dataset auto-download via the HF mirror.** `princeton-nlp/SWE-bench_Verified`
   must download through `hf_endpoint` (hf-mirror.com), not hang on huggingface.co.
   Check the controller logs for the download lines.
3. **Oracle → resolved == N.** Gold-patch replay must resolve **all 3** instances
   (`swe-bench.resolved == 1.0`). Anything less is an environment or harness problem,
   never "agent quality".
4. **Docker image footprint.** The harness builds per-instance images (GiBs each).
   `docker system df` on the driver must show comfortable headroom on the 120 GiB disk.
5. **`sys.executable` resolution.** The harness is spawned as
   `sys.executable -m swebench.harness.run_evaluation` — on the controller this is
   the SYSTEM python3.11 that cloud-init installed (there is no venv). Confirm
   `swebench` imports there: it is installed via the plan-driven extras (any
   `suite:` task in the plan adds the `[swebench]` extra to the controller deps).
6. **Placement stays local.** `driver.placement == "local"` on the driver host —
   the harness containers run where Docker runs; the suite hard-rejects anything else.
7. **Container naming matches teardown.** Harness containers are named
   `sweb.eval.<instance_id>.<run_id>`; suite teardown removes containers via
   `docker ps --filter name=<run_id>`. Verify the run\_id appears in the live container
   names so cleanup catches exactly this run's containers and nothing else.
8. **Harness stderr surfacing.** Force one failure (or inspect any real one): the
   error must carry the last 2000 chars of harness stderr — a real cause, not a bare
   exit code.

## cn-region gotchas

* **Docker Hub reachability is auto-handled.** The driver detects it at boot and
  routes accordingly (direct / ACR / fail-loud); there is no per-account mirror
  address to configure. The Aliyun image accelerator (`<id>.mirror.aliyuncs.com`)
  deliberately is NOT used — it has no OpenAPI, so a tool cannot obtain it; the
  account's ACR endpoint (`get_instance_endpoint`) is used instead when needed.
  A mainland region also walls off GitHub (repo clones): if the driver reports a
  blocked strategy, run it in a region with direct egress or pre-stage images.
* **huggingface.co is unreliable.** Keep `driver.hf_endpoint: https://hf-mirror.com`
  for the dataset auto-download.
* **NAT gateway + EIP bill per hour.** Fetch, then tear down promptly — do not leave
  a finished campaign's infra idling overnight.
* **FC cold start is \~86 s.** Amortize it by batching several tasks into one campaign
  instead of one campaign per task.

## Expected outcomes

| mode | `swe-bench.resolved` | `swe-bench.cost_per_resolved` | `provenance.scaffold` |
| - | - | - | - |
| `oracle` | `== 1.0` (3/3) | absent — zero LLM spend | `oracle@slice2` |
| `llm` | `>= 0` (real agent, may miss) | **real** value from actual usage events | `qwen-llm@slice2` |

## Reading the results

`csbench fetch` writes one JSON record per task (plus parquet sidecars where a task
produces series) into `--dest`. Then:

```bash theme={null}
# measurements as a table (needs the [store] extra)
csbench query --table measurements --results results/swe-smoke --format table

# stage-tree trace of the run
csbench trace list --results results/swe-smoke
# <run_id> here is the RECORD's run id (from `csbench trace list`), not the
# harness `csbench-<8hex>` id used in container names.
csbench trace show <run_id> --results results/swe-smoke
```

Look for: measurement keys `swe-bench.resolved` and (llm mode) `swe-bench.cost_per_resolved`;
`provenance.suite_id == "swe-bench"`; `provenance.scaffold` as in the table above. The raw
record (`suite:swe-bench.json`) also carries the artifact manifest — predictions, results,
trajectory, and usage files with sha256 hashes.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.