Documentation Index
Fetch the complete documentation index at: /llms.txt
Use this file to discover all available pages before exploring further.
把任意 Clousight Bench 套件作为原生 pytest 测试或 CI 退出码门禁运行,带阈值。
assert_runclousight-bench 后,pytest 插件经 pytest11 entry point 自动加载,无需
conftest 配置。在任意测试里调 assert_run:它跑一个套件,除非运行完成且所有阈值达标,
否则抛 AssertionError:
from clousight_bench.pytest_plugin import assert_run
def test_our_llm_endpoint_mmlu_gate():
assert_run(
domain="llm", suite="mmlu", platform="llm-endpoint",
target={"endpoint": "https://.../v1", "model": "qwen-max",
"credentials_ref": "env:DASHSCOPE_API_KEY"},
params={"limit": 100},
thresholds={"mmlu.accuracy": {"min": 0.75},
"mmlu.avg_latency_ms": {"max": 3000}},
)
clousight fixture:
def test_offline_gate(clousight):
clousight(domain="transactional-db", suite="tpc-c", platform="benchbase-local",
thresholds={"tpc-c.throughput_req_per_sec": {"min": 500}})
target 默认 {"mode": "mock"}(离线路径),CI 里无云账号也能跑门禁;指向真实端点/服务即可门禁生产。
csbench run --assertcat > gate.yaml <<'YAML'
thresholds:
mmlu.accuracy: {min: 0.75}
mmlu.avg_latency_ms: {max: 3000}
YAML
csbench run --domain llm --benchmark mmlu --platform llm-endpoint \
--config prod.yaml --assert gate.yaml
# 运行未完成或任一阈值不达 → 退出码非零
| 形式 | 含义 |
|---|---|
{key: {min: x}} | 值必须 ≥ x |
{key: {max: y}} | 值必须 ≤ y |
{key: x} | {min: x} 简写 |
assert_run(pytest)与
csbench run --assert(CI 退出码)。