Skip to main content
The core orchestrates a fixed lifecycle; everything product-specific is a plugin. This page names the load-bearing types and the entry points that register them. For line-level code questions, use the in-page Ask assistant or DeepWiki. Import the public contracts + data model from clousight_bench (or clousight_bench.api) — never from clousight_bench.core.* (internal). The public surface is what the plugin-API version (clousight_bench.PLUGIN_API_VERSION) governs.

​
Result contract

A run emits a ResultRecord (schema 0.4): status (completed / failed / invalid / unsupported), measurements (each with value, unit, reproducibility_class, official, optional ci), optional per-item items (ItemResults with 4-state ItemScores the measurements aggregate), findings (stable code, severity), and fingerprints (benchmark / environment / implementation / record_digest). See Architecture for the full field table.

​
Plugin contracts

The primary public unit is a benchmark — a BenchmarkSuite + Evaluator (see Add a benchmark). Since plugin API 2.0 the suite/evaluator pair is the only benchmark rail (the old Task contract is gone); the current PLUGIN_API_VERSION is "3.0" (the OTel-native tracing break).

​
Entry points

Third-party (and commercial) packages extend the harness by declaring these entry points (see pyproject.toml):
  • clousight_bench.benchmark_suites — benchmark suites
  • clousight_bench.evaluators — evaluators (namespaced measurements + items())
  • clousight_bench.metrics — composable add-on metrics
  • clousight_bench.judges — LLM-as-judge providers
  • clousight_bench.domains — domain packs (tasks + platform adapters)
  • clousight_bench.runtime_providers — cloud runtime providers
  • clousight_bench.resource_reapers — orphan-resource reapers
  • clousight_bench.enrichers — post-run enrichers (e.g. pricing)
  • clousight_bench.span_exporters — trace span exporters
  • clousight_bench.asset_resolvers — private/gated dataset resolvers
See Add a benchmark and Plugins for worked examples.