Gate on engine quality in CI

Once an agent engine is part of your product, it can break like any other dependency. But it breaks quietly — there's no stack trace. A prompt change, a model swap, a settings edit, a new release from your provider: the turn still exits with no error, still prints something that sounds right, and quietly does less work than it did last week.

This guide sets up a CI job that catches that. It runs stella and Claude Code on the same set of committed tasks on every pull request. The important part: it treats their two results as answers to two different questions.

Two kinds of check

Most attempts at this fail the same way. Someone builds a test harness, scores both engines, gets a number, and sets a threshold on that number. Within a month, the check gets turned off because it fails on days when nothing actually changed.

"Did the engine work" and "was the answer good" are two different questions. Only one of them gives a steady enough answer to block a merge on.

Everything below is built on that split between the two questions. If you remember one thing from this page, remember that.

Before you start

Step 1: Commit the task set

Keep the list small, fixed, and checked into the repository. Five to ten tasks is plenty. This gate is only looking for a loop that stopped working, and a broken loop shows up on the very first task.

ci/engine-tasks/
  001-add-pagination/
    TASK.md              # the prompt, verbatim
    verify.sh            # exit 0 if the change is correct
  002-fix-null-deref/
  003-rename-across-files/

What makes a good task

Step 2: Run both engines headless

Both engines take a prompt and print JSON you can parse. The JSON has a different shape for each one, and that difference trips people up — so read this part carefully.

Give each engine its own copy of the code, so their edits can't see each other, and each copy can be graded on its own:

task=ci/engine-tasks/001-add-pagination
for e in stella claude; do rm -rf "work/$e"; git worktree add -f "work/$e" HEAD; done

# Stella — one JSON object on stdout.
( cd work/stella && stella --spend-limit 1.00 \
  run --output-format json "$(cat "../../$task/TASK.md")" ) > work/stella/stella.json

# Claude Code — a JSON *array*; the result is the last element.
( cd work/claude && claude -p --output-format json --permission-mode acceptEdits \
  "$(cat "../../$task/TASK.md")" ) > work/claude/claude.json

Both engines need to be able to edit files without a person approving each change. --permission-mode acceptEdits does that for Claude Code. Check claude --help on the exact version you pinned, rather than copying a flag list from somewhere else — these flags change between releases.

Here's what each one gives you, cut down to the keys your gate actually reads:

// stella.json — see /docs/scripting for the full envelope contract
{
  "schema_version": 1,
  "status": "completed",        // or verification_failed | aborted | error
  "text": "…",
  "reason": null,               // non-null whenever status is not completed
  "cost_usd": 0.184,
  "model": "anthropic/claude-fable-5",   // what actually ran, not what you asked for
  "events": [ { "type": "tool_start", "call": { "name": "task_create" } }, … ]
}

// claude.json — the last array element
{
  "type": "result",
  "subtype": "success",
  "is_error": false,
  "num_turns": 14,
  "result": "…",
  "total_cost_usd": 0.211,
  "duration_ms": 48213,
  "permission_denials": [],     // a non-empty array here is a loop-health signal
  "terminal_reason": "completed"
}

Read the model field, not the flag you passed. stella reports the model that actually ran the worker turns. A worker-model route can swap out the model you asked for with --model. Always attribute spend to what the receipt says, not to your own flag.

files_touched isn't in the same place on every run. A plain (default) run puts it at the top level, as files_touched. A --pipeline <variant> run also adds task_class, verdict, revisions, and candidates_run — but its file list sits one level deeper, at .files_touched.files_touched. Counting tool_start events works the same way on both, which is why the checks below use events instead.

Step 3: Loop correctness

This is the same gate stella runs on itself. The vocabulary is worth reusing, since it comes from watching real failures happen. loop-bench sorts every run into one of four results, checked in this order:

The order matters. A solved task always outranks everything else — if the task got solved, work happened, by definition, no matter what the event log is missing. The two results that fail the gate are the two where nothing happened at all: loop_broken means zero_work && reward != 1.0.

Applied to the receipts from step 2, that becomes four checks:

#!/usr/bin/env bash
# ci/assert-loop-health.sh — the deterministic half. No model judges anything.
set -euo pipefail

fail() { echo "loop-broken: $1" >&2; exit 1; }

s=work/stella/stella.json
c=work/claude/claude.json
[ -s "$s" ] || fail "stella emitted no receipt"
[ -s "$c" ] || fail "claude code emitted no receipt"

jq -e 'if .schema_version == 1 then . else error("unsupported schema_version") end' \
  "$s" > /dev/null || fail "stella envelope version changed under us"

tools=$(jq '[.events[] | select(.type == "tool_start")] | length' "$s")
[ "$tools" -gt 0 ] || fail "stella made zero tool calls on an editing task"

jq -e '.status == "completed" or (.reason != null)' "$s" > /dev/null \
  || fail "stella neither completed nor said why"

result=$(jq '[.[] | select(.type == "result")] | .[0]' "$c")
[ "$result" != "null" ] || fail "claude code produced no result element"

denials=$(jq '.permission_denials | length' <<<"$result")
[ "$denials" -eq 0 ] || fail "claude code was blocked by $denials permission denial(s)"

jq -e '.terminal_reason != null' <<<"$result" > /dev/null \
  || fail "claude code ended without a terminal reason"

echo "loop healthy: $tools tool calls, clean termination"

Notice what's missing from that script: any check on whether the task got solved. A task nobody solves can still pass this gate, as long as the loop actually ran. That's the point. You're gating on the machine, not on the model.

Step 4: The quality delta

This part gets reported, not enforced. Same two receipts as before, but now read for the outcome instead of the health check.

Each engine ran in its own copy of the code. That's what makes them comparable at all, and each one has its own workspace to grade.

#!/usr/bin/env bash
# ci/quality-delta.sh — writes a summary, never exits non-zero.
set -uo pipefail
task=001-add-pagination

graded() {  # $1 = the workspace that engine worked in
  ( cd "$1" && "./ci/engine-tasks/$task/verify.sh" >/dev/null 2>&1 ) && echo yes || echo no
}

stella_cost=$(jq -r '.cost_usd' work/stella/stella.json)
claude_cost=$(jq -r '[.[] | select(.type == "result")] | .[0].total_cost_usd' \
                 work/claude/claude.json)

{
  echo "| engine | solved | cost | model |"
  echo "|---|---|---|---|"
  echo "| stella | $(graded work/stella) | \$$stella_cost | $(jq -r '.model' work/stella/stella.json) |"
  echo "| claude code | $(graded work/claude) | \$$claude_cost | $CLAUDE_MODEL |"
} >> "$GITHUB_STEP_SUMMARY"

Don't set a hard threshold based on that table. With only five tasks, one task flipping changes the score by twenty points, and that much random swing is completely normal. The table's job is to show a trend across many pull requests. When it moves the same direction again and again, let a person decide what that means.

The workflow

name: engine-gate
on: pull_request          # never pull_request_target — that hands forks your secrets

jobs:
  gate:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4

      - name: Does this PR touch anything the engine depends on?
        id: scope
        run: |
          if git diff --name-only "${{ github.event.pull_request.base.sha }}"...HEAD \
             | grep -Eq '^(prompts/|src/agent/|ci/engine-tasks/)'; then
            echo "changed=true" >> "$GITHUB_OUTPUT"
          else
            echo "changed=false" >> "$GITHUB_OUTPUT"
            echo "No engine-affecting changes — suites skipped." >> "$GITHUB_STEP_SUMMARY"
          fi

      - name: Loop health — the blocking half
        if: steps.scope.outputs.changed == 'true'
        run: ./ci/run-both.sh && ./ci/assert-loop-health.sh

      - name: Quality delta — advisory
        if: steps.scope.outputs.changed == 'true'
        continue-on-error: true
        run: ./ci/quality-delta.sh

Mark it as a required status check. A check that isn't required can't block a merge, and a check that can't block a merge doesn't do anything. This was learned the hard way: a dependency update once broke the benchmark workflow, the workflow reported failure, and the pull request merged twelve minutes later anyway. It stayed broken for five days.

Filter the scope inside a step, not in the trigger. If you filter with on.pull_request.paths, the workflow doesn't run at all when nothing matches — so it never reports a result, and a required check that never reports blocks the pull request forever. The job above always runs and always reports something. Only the expensive part is conditional. An unrelated pull request just pays for one runner starting up, not for two full agent runs.

Changing stella itself

The harness above works for any product using stella. But if you're changing the engine itself, stella has its own version built in, and it's cheaper and more precise:

cargo run -p loop-bench -- --n 4                     # four tasks, the default pool
cargo run -p loop-bench -- --json-out report.json    # table on stdout, report on disk
cargo run -p loop-bench -- --analyze-only --jobs-dir <dir> --job-name <name>   # free

It reads stella-events.jsonl for each trial, sorts each one into the same four results as above, and exits with code 1 if any trial comes back loop_broken — even if other trials passed. Exit code 2 means the task list came up empty, or the report couldn't be written. Exit code 3 means no trial results were found at all. That's an infrastructure problem, not a loop problem. Exit code 4 means the loop worked fine, but fewer trials passed than --min-pass required. Exit code 8 means more of the trials died than finished. A verdict over what is left is a verdict about a night that mostly did not run.

stella runs this nightly, not on every pull request, since it needs Docker, a task runner, a provider key, and real money. Two details of that setup are worth copying. First, the task list is pinned by name, not just "the first four in the list" — since the list order can change and would silently change what gets measured. Second, the pass-rate floor ships turned off at first. It only gets turned on once the job's own reports show what normal looks like. A floor picked by guesswork on a cheap model fails every single night, and a gate that always fails is a gate nobody reads.

What this cannot tell you

A comparison harness that oversells itself is worse than no harness at all.

See also