A metric scores one item at a time and defines how its per-item scores
aggregate. Metrics are decoupled from suites: a run can apply several over the same
items (the suite’s objective score lives in its evaluator; add-ons — an
answered-rate, a cost-efficiency, an LLM-judge rubric — plug in
without forking the evaluator).
Requires a suite that emits per-item items (Evaluator.items(raw)), since
metrics score items. The bundled mmlu / gsm8k / human-eval suites do.
Run a metric
Bind it on the evaluator, or opt in per run:
A metric emits measurements under <suite_id>.<metric_id> and appends a per-item
ItemScore (4-state: ok / fail / skip / error) to every item. Metrics are
isolated: one metric raising on an item is recorded as status="error" and
the run continues — it never voids the run or another metric.
The reference metric: answered-rate
Deterministic, judge-free: the fraction of items the SUT actually answered (a
non-null output). answered-rate is its entry-point name; its metric id
is answered_rate — the id is what goes in extra_metrics and in measurement
keys. Bound to mmlu / gsm8k out of the box → they also emit
mmlu.answered_rate / gsm8k.answered_rate. A model can be accurate on what it
answers yet answer little.
Write your own
Register it and bind or opt-in:
required_inputs are validated per item: an item missing a required field is
scored skip (excluded from the aggregate), never a crash. Return None from
score_item to leave an item unscored by this metric.
Aggregation
The default aggregate() uses core.aggregate — ratio/mean (with a Wilson or
normal confidence interval), geomean, percentiles — over the metric’s ok/fail
scores; skip/error are excluded from the denominator. Override aggregate()
for custom shapes, or emit per-group breakdowns (e.g. MMLU by subject). 