Skip to main content
Clousight Bench is a core engine + entry-point plugins. The engine knows how to run a benchmark, attribute provenance, and persist schema-0.4 records; it knows nothing about any specific cloud, task, or dataset. Everything domain-specific — benchmark suites, platform adapters, enrichers, reapers — is a plugin discovered at runtime through Python entry points. The guiding principle: customize by writing a plugin, never by forking the core. The reference domains ship in the package so it works out of the box and so you have a canonical example to copy; your own logic lives in your own package.

​
The layers

Because the reference domains are themselves registered through the entry-point system, there is nothing special about them — your plugins are first-class in exactly the same way.

​
The entry-point groups

Register a plugin by advertising an entry point in your package’s pyproject.toml. The engine discovers them with importlib.metadata.entry_points (clousight_bench.core.registry): Adding a benchmark = implement a BenchmarkSuite + Evaluator (optionally Metric / judge), register those entry points, and run it with csbench run --benchmark <id>. This is the one public benchmark contract — for open-source and commercial suites alike (a commercial suite ships the same entry points from a private package). Since plugin API 2.0 the suite/evaluator pair is the ONLY benchmark rail — the old internal Task contract is gone. The current plugin API is 3.0 (the OTel-native tracing break; the registry refuses plugins pinned to a 2.x range). The scorer triangle — three scorers, three distinct jobs:
  • Evaluator — the suite’s official verdict (artifacts → measurements); the never-re-score carrier. One per suite, namespaced <suite_id>.*.
  • Metric — additive, composable per-item dimensions layered over the evaluator’s items(), emitted as <suite_id>.<metric_id>.
  • Judge — an implementation detail a Metric may use (LLM-as-judge via the clousight_bench.judges seam); never a third scoring path of its own.
The adapter’s status literal (skeleton / reference / …) is not flipped by installing a plugin: a skeleton adapter becomes runnable in real mode purely because get_runtime_provider(<provider>) now returns a non-None plugin — i.e. installing the provider plugin wires it, without editing the adapter.

​
Write your own plugin

The primary extension is a benchmark — a BenchmarkSuite + Evaluator, run with csbench run --benchmark <id>. That has its own worked, copy-pasteable walkthrough:

Add a benchmark

Implement a Suite + Evaluator, register two entry points, run it.
Add-on metrics and judges follow the same pattern (a class + an entry point under clousight_bench.metrics / .judges). Everything ships from your own package’s pyproject.toml entry points — you never touch the installed clousight-bench. A commercial suite/metric/judge/dataset is just a private package that registers the same entry points; install it and csbench list picks it up, requires_plugin_api guarding compatibility. DomainPack + ProviderAdapter remain the way to add a whole product category and its platform adapters (e.g. a new cloud); benchmarks themselves always land as BenchmarkSuite + Evaluator entry points.

​
Why the reference logic is packaged, not download-only

  • Out of the box: pip install clousight-bench and you can run agent-runtime immediately — no assembling parts.
  • It is the template: the reference domains are the best worked example of a well-formed plugin; you copy their shape for your own.
  • One consistent chain: the data-plane probe ships in the core package too, so an in-region probe carrier just pip install clousight-bench[probe] and has the same probe logic the control plane uses.
  • The source is fully open on GitHub for reading, forking, and PRs — but the primary customization path is a plugin package, not editing installed source.