stella tune
Eval-driven self-tuning of Stella's own policy — A/B one knob over two loop-bench runs, promote the winner only when it confidently wins, and roll it back on the record.
stella tune is how Stella's own policy gets tuned by evidence instead of taste. It A/B tests one knob over two loop-bench runs, scores each arm from the per-trial rewards, and promotes the candidate only when it confidently beats the baseline — with a reversible rollback record either way.
The first knob is the worker's reasoning effort. Offline: reads two result files and the local ledger, writes settings only on --promote. No provider, no API key.
Synopsis
stella tune effort --baseline <file> --candidate <file> \
[--baseline-effort <effort>] [--candidate-effort <effort>] \
[--promote] [--scope <project|user>] [--min-samples <n>]
stella tune rollback
stella tune statusWhat it does
Most "tuning" of an agent is somebody changing a default because a run felt better. That is unfalsifiable. The flow here is deliberately boring: run a fixed held-out task list twice, once per arm, and let the numbers decide.
# 1. Run the held-out task list at each effort, capturing --json
loop-bench --json > base.json # worker effort: medium
loop-bench --json > cand.json # worker effort: high
# 2. Score the arms — reports only, changes nothing
stella tune effort --baseline base.json --candidate cand.json
# 3. Apply the winner, with a rollback record
stella tune effort --baseline base.json --candidate cand.json --promotestella tune effort
A/B the worker reasoning-effort knob over two loop-bench --json result files and select the winner.
--baseline <file>Requiredloop-bench --json output for the baseline arm.
--candidate <file>Requiredloop-bench --json output for the candidate arm.
--baseline-effort <effort>The worker effort the baseline arm ran at.
Default medium
--candidate-effort <effort>The worker effort the candidate arm ran at.
Default high
--promoteApply the winner to settings. Omitted, the command reports and records the A/B receipt but changes nothing.
Default off
--scope <project|user>Which settings scope to write on --promote.
Default project
--min-samples <n>Minimum trials required in every arm before any promotion is allowed.
Default 5
--min-samples is a floor on promotion, not on reporting. A two-trial A/B still prints its comparison — it just cannot move your settings on the strength of it.
stella tune rollback
Revert the last effort promotion, restoring the prior worker effort. Every promotion writes the record that makes this possible, so a tuning decision is never one-way.
stella tune rollbackstella tune status
Show the self-tuning ledger — the experiments that have run, and the promotion currently in effect.
stella tune statusWhy promotion is conservative
A/B testing an agent is noisy: the same task list at the same settings does not return the same reward twice. Promoting on a bare mean would ratchet the configuration around on noise, and every such move is invisible to you afterward. So promotion requires the candidate to confidently win, --min-samples sets a hard floor under both arms, and the write is recorded with a rollback — the three things that make an automatic policy change safe to leave switched on.
--promote writes to your settings.json at the chosen scope, and it reports the prior value so you can see what it displaced. Run without it first and read the comparison; the receipt is recorded either way, so nothing is lost by looking.
One promotion side effect is worth knowing: if effort_auto was on, promoting turns it off, because an automatic effort selector would otherwise override the pinned value the A/B just chose. stella tune rollback restores both the prior effort and effort_auto, so the pair is reverted together rather than leaving you pinned with the selector still disabled.
Related
stella proposals
Review what the adaptive-context loop wants to make durable before it lands — the proposals it induced, the evidence behind each, and Keep/Edit/Ignore as immutable events.
stella scoreboard
What the work cost, and whether anyone said it was good — calls, characters typed, follow-ups, and the verdict a merged or closed pull request implies. No model judges anything.