SWE-bench live smoke runbook

​

Preconditions

• Aliyun balance ≥ ¥100. Postpaid ECS creation is rejected below this account balance — the submit will fail at terraform apply with an opaque quota-like error.
• RAM credentials. MAIN-account AK exported for terraform (ALICLOUD_ACCESS_KEY / ALICLOUD_SECRET_KEY / ALICLOUD_REGION); the controller’s own SDK calls use the sub-account instance role that controller.tf provisions.
• configs/agent-runtime-aliyun.local.yaml (gitignored, never committed) with target.region, target.blob_bucket, the credential env-var names under target.auth_env, and target.arms_license_key for telemetry.
• target.mode: real in that local config. Without it the suite bridge defaults to mock — you would pay for a full-price ECS campaign that replays canned fixtures (csbench submit prints a loud warning if a suite: task is submitted non-real).
• ARMS license key present in that local config (trace/metric export).
• llm mode only — key delivery is NOT yet wired for prod submit. DASHSCOPE_API_KEY must be present in the DRIVER host’s environment at provision time: the SUT client forwards it into the CreateAgentRuntime provision env, but prod cloud-init does not deliver it to the driver today. Two working options: (i) submit with controller_debug=true, SSH to the controller and export DASHSCOPE_API_KEY=... into the orchestrator’s environment before the campaign claims the suite task; or (ii) run llm mode from a locally-driven (non-submit) execution where your own shell environment carries the key. Oracle mode needs no LLM key at all.

​

The plan

version: "1"
domain: agent-runtime
platform: aliyun-agentrun

# Campaign-wide hard cap in USD; the controller stops before provisioning a
# billable task that would cross it.
cost_budget: 10.0

# The controller ECS doubles as the docker-capable SWE-bench driver host.
# Every key maps to a `-var controller_*` terraform var; unknown keys are
# rejected at submit time.
driver:
  install_docker: true          # the upstream harness runs docker on the driver
  system_disk_size: 120         # GiB — 3 instance images + base easily exceed the 40 default
  # NO mirror address needed: the driver auto-detects its image strategy at boot
  # (probes Docker Hub; direct-pull when reachable, ACR when blocked, fail-loud
  # otherwise — region-agnostic, see driver_image.py). Uncomment only to FORCE a
  # specific mirror:
  #   docker_registry_mirror: "https://<your-own-mirror>"
  # huggingface.co is unreliable from cn regions — pin the mirror for the
  # SWE-bench_Verified dataset auto-download.
  hf_endpoint: "https://hf-mirror.com"
  # image builds need ≥4c8g — the default e-c1m2.large is too small for the harness.
  instance_type: "ecs.c6.xlarge"

tasks:
  - task_id: "suite:swe-bench"
    params:
      # oracle = gold-patch replay through the REAL AgentRun SUT + REAL harness:
      # proves the whole pipe with zero LLM spend. Expected: resolved == 1.0.
      # Switch to `llm` for the paid Qwen variant (needs DASHSCOPE_API_KEY on
      # the driver; yields a real swe-bench.cost_per_resolved).
      agent_kind: oracle
      # The 3 bundled real SWE-bench Verified rows
      # (src/clousight_bench/suites/swe_bench/fixtures/instances_subset.json).
      instance_ids:
        - django__django-11099
        - sympy__sympy-20590
        - pytest-dev__pytest-7205

​

Command sequence

# 1. Submit — writes the launch spec to OSS, terraform-applies controller + NAT.
csbench submit configs/swe-bench-smoke.plan.yaml \
  --config configs/agent-runtime-aliyun.local.yaml \
  --watchdog-timeout 5400
# prints the campaign_id, e.g. camp-ab12cd34

# 2. The laptop may go offline. Poll progress (reads OSS only):
csbench status  camp-ab12cd34 --config configs/agent-runtime-aliyun.local.yaml
csbench logs    camp-ab12cd34 --config configs/agent-runtime-aliyun.local.yaml

# 3. When status shows done, pull the records:
csbench fetch   camp-ab12cd34 --config configs/agent-runtime-aliyun.local.yaml \
  --dest results/swe-smoke

# 4. Backstop cleanup — ALWAYS run if self-destruct is not confirmed:
csbench teardown camp-ab12cd34 --config configs/agent-runtime-aliyun.local.yaml

​

Live-verification checklist (8 points)

1. Harness report filename + keys. With installed swebench>=3.0 the report is written to <report_dir>/<model_name>.<run_id>.json (slashes in the model name become __) with top-level keys resolved_instances / total_instances / resolved_ids. The suite fails loudly on schema drift — a clean pass here pins the version as compatible.
2. Dataset auto-download via the HF mirror. princeton-nlp/SWE-bench_Verified must download through hf_endpoint (hf-mirror.com), not hang on huggingface.co. Check the controller logs for the download lines.
3. Oracle → resolved == N. Gold-patch replay must resolve all 3 instances (swe-bench.resolved == 1.0). Anything less is an environment or harness problem, never “agent quality”.
4. Docker image footprint. The harness builds per-instance images (GiBs each). docker system df on the driver must show comfortable headroom on the 120 GiB disk.
5. sys.executable resolution. The harness is spawned as sys.executable -m swebench.harness.run_evaluation — on the controller this is the SYSTEM python3.11 that cloud-init installed (there is no venv). Confirm swebench imports there: it is installed via the plan-driven extras (any suite: task in the plan adds the [swebench] extra to the controller deps).
6. Placement stays local. driver.placement == "local" on the driver host — the harness containers run where Docker runs; the suite hard-rejects anything else.
7. Container naming matches teardown. Harness containers are named sweb.eval.<instance_id>.<run_id>; suite teardown removes containers via docker ps --filter name=<run_id>. Verify the run_id appears in the live container names so cleanup catches exactly this run’s containers and nothing else.
8. Harness stderr surfacing. Force one failure (or inspect any real one): the error must carry the last 2000 chars of harness stderr — a real cause, not a bare exit code.

​

cn-region gotchas

• Docker Hub reachability is auto-handled. The driver detects it at boot and routes accordingly (direct / ACR / fail-loud); there is no per-account mirror address to configure. The Aliyun image accelerator (<id>.mirror.aliyuncs.com) deliberately is NOT used — it has no OpenAPI, so a tool cannot obtain it; the account’s ACR endpoint (get_instance_endpoint) is used instead when needed. A mainland region also walls off GitHub (repo clones): if the driver reports a blocked strategy, run it in a region with direct egress or pre-stage images.
• huggingface.co is unreliable. Keep driver.hf_endpoint: https://hf-mirror.com for the dataset auto-download.
• NAT gateway + EIP bill per hour. Fetch, then tear down promptly — do not leave a finished campaign’s infra idling overnight.
• FC cold start is ~86 s. Amortize it by batching several tasks into one campaign instead of one campaign per task.

​

Expected outcomes

​

Reading the results

# measurements as a table (needs the [store] extra)
csbench query --table measurements --results results/swe-smoke --format table

# stage-tree trace of the run
csbench trace list --results results/swe-smoke
# <run_id> here is the RECORD's run id (from `csbench trace list`), not the
# harness `csbench-<8hex>` id used in container names.
csbench trace show <run_id> --results results/swe-smoke