stella calibration

Answer "how often does the judge approve work that CI later rejects?" from what the workspace has already recorded. Every verification verdict is persisted in the local event journal with the ladder snapshot it was decided from, and every PR/CI observation lands in the same stream — so calibration is a reading, not a new telemetry pipe. Entirely local; no API key, no network, and it never writes.

Synopsis

stella calibration                # human-readable report
stella calibration --format json  # machine-readable tallies

What it reports

Two cohorts, deliberately side by side — a false-positive rate means nothing without the cohort it should be compared against:

• model judge — JudgeVerdict events with passed: true from a model judge (the ladder escalated on inconclusive evidence).
• deterministic — passes the ladder credited itself (fail→pass flip, green tests, budget, and every pre-submit audit).

For each: how many passes were recorded, how many were reconciled — a terminal CI verdict (Passing/Failing) was observed later in the same session's stream — and how many of those reconciled passes failed CI. The rate is printed only over reconciled passes; with no CI ground truth yet, it is reported as unmeasured, never as zero.

Honest limits

• Reconciliation is stream-local and forward-only: a session's terminal CI verdict covers the passes recorded before it, and passes after the last CI observation stay unreconciled.
• Pending/Running CI states reconcile nothing — absence of a verdict is not a verdict.
• Reverts and rollbacks are not yet a ground-truth source; the report says so rather than approximating.

See also

• stella inspect — the exact context any past call was sent
• stella stats — cost and resolve-rate per provider/model