stella daemon

A long-running command started from a terminal — run, goal, monitor, fleet — is handed to a supervisor. The work becomes a detached process that survives the window closing, an ssh disconnect, and a logout. The terminal you started it in stays open only to stream its output. stella daemon is how you find that process again afterward.

Synopsis

stella run "…" --detach
stella daemon list
stella daemon attach [id]
stella daemon logs [id] [-n LINES]
stella daemon stop <id>
stella daemon resume [id]
stella daemon resume-all [--dry-run] [--ceiling MINUTES]
stella daemon install [--label NAME] [--keep-alive] -- <stella args…>
stella daemon install --resume-all [--label NAME]
stella daemon uninstall <label>

Why runs are supervised

Closing a terminal sends SIGHUP to its foreground process group. Without supervision, that would kill the run mid-turn: no answer, no record of why it stopped, and no way back to it. A supervised run starts its own session before it starts work, so the hangup never reaches it.

You do not need to start a supervised run yourself. Every terminal run already is one. The first two lines it prints say so:

▸ supervised ses-1785956826121-25167 — survives this terminal closing
  reattach with stella daemon attach ses-1785956826121-25167

Close that terminal, open another, and the run is still there:

$ stella daemon list
ID                           STATUS       WHAT
ses-1785956826121-25167      Running      stella: migrate the config loader to serde

$ stella daemon attach ses-1785956826121

What supervision survives

It survives the terminal closing: a closed window, a dropped ssh session, a logout. The run never notices, because it left that terminal's session before it began.

A killed process — an out-of-memory kill, kill -9, a reboot — is a bigger failure, but it's recoverable too. Every turn checkpoints into the workspace's durable work journal at each completed step, so a run killed mid-turn leaves a point to resume from, something a closed window never needed. resume picks it up. (For interactive deck sessions, stella resume is the command to use — it reopens a conversation. daemon resume continues a headless run's interrupted turn.)

Which invocations are supervised

Only ones with a terminal to lose. A run with no terminal is already safe from a hangup, so supervising it would just add overhead for nothing. Specifically:

--foreground (or STELLA_FOREGROUND=1) opts any invocation out.

--detach: start it and get the prompt back

Supervision keeps the run alive when the terminal goes away. It doesn't hand the terminal back while the run keeps going. The launching process stays open for the whole run, streaming the console, and Ctrl-C there stops the run, the same as it does everywhere else.

--detach is the other half. The work is supervised exactly as it would be otherwise (same session id, same console, the whole stella daemon surface can address it), but the launcher returns as soon as the run is registered:

$ stella run "migrate the config loader to serde" --detach
▸ detached ses-1785956826121-25167 — running in the background
  follow it with stella daemon attach ses-1785956826121-25167
  stop it with   stella daemon stop ses-1785956826121-25167
$

Two differences from an attached launch, both intentional:

• A terminal is not required. Plain supervision does nothing without one, since a run with no terminal to lose is already safe. --detach forces supervision instead of only allowing it, because "return right away" is just as useful from a pipe, a container, or a cron job as from a terminal.
• The exit code belongs to the launch, not the run. stella run --detach; echo $? tells you whether the launch succeeded. The run's own result shows up later in stella daemon list.

Detaching from a script

The banner above prints to stderr, so it never shows up in a pipe. Under --output-format json or stream-json, stdout carries one launch summary naming the session, the id a script needs before it can do anything with the run it just started:

$ stella run "migrate the config loader to serde" --detach --output-format json
{
  "schema_version": 1,
  "status": "detached",
  "session_id": "ses-1785956826121-25167",
  "attach": "stella daemon attach ses-1785956826121-25167",
  "stop": "stella daemon stop ses-1785956826121-25167"
}

id=$(stella run "…" --detach --output-format json | jq -r .session_id)
stella daemon stop "$id"

stream-json returns the same object, compact, on one line. status reads detached rather than ok: the launch succeeded, and whether the run itself succeeds is a separate, later question. See scripting for the full family of output formats.

This applies to every supervised command: run, goal, monitor, fleet. STELLA_DETACH=1 turns it on for a whole shell. --foreground wins if both are given, since the flag asking for less machinery is the safer choice when the command line contradicts itself.

A detached run can't ask this terminal anything, since nothing is attached anymore, but it doesn't lose the answer either. A plan that crosses the scope thresholds parks and waits, as described below, and the next stella daemon attach asks the question.

Scope review while supervised

A supervised run's input is a file and its console is a log, so it can't show a scope-review prompt the usual way. But supervision doesn't cost you the answer. When a stella run plan crosses the scope thresholds, the run parks: the proposal is written into the session's side files, stella daemon list shows the run as Needs Input, and whichever terminal is attached asks the question, whether that's the terminal that launched it (if still open) or a later stella daemon attach. The prompt appears on the attached terminal's stderr, so --output-format json stdout stays readable by scripts. The same answers work as at a normal prompt: y approves, n (or an empty line) cancels, anything else is a note the planner uses to revise the plan.

A parked run holds its budget and its workspace until someone answers. It's visible in list the whole time, and stella daemon stop (or any SIGTERM) cancels it cleanly. There's no setting to skip this review.

list

Every supervised run on this machine, newest first. This only reads local data. It needs no API key and doesn't need to check with any provider, so a run whose model configuration is now broken is still visible and stoppable.

STATUS is based on a lock the run holds for its whole life, not on its process id, because the operating system releases that lock as soon as the process dies for any reason (crash, kill, or power loss included). A run whose process is gone without recording a result shows as Crashed. One that left a resume point is marked Crashed ↩, with the resume command to use shown under the table. A run parked on a scope review shows as Needs Input.

attach

Streams the run's output into this terminal, from the beginning, and stays until the run ends. A run that already finished prints its full output and exits. The id can be any unique prefix, or the run's process id, which is what ps, top, Activity Monitor, and an out-of-memory kill log give you. Omit the id to use the most recently started run. If the run is parked on a scope review, attach shows the proposal and sends your answer.

Detaching again with Ctrl-C leaves the run running. stella daemon stop is what actually stops it.

logs

Shows the tail of a finished or running console, without following it live.

stdout and stderr are kept in two separate files and replayed onto the two streams they came from, so stella run --output-format json stays readable by scripts even through a supervisor. A small write index is kept beside the console, and logs uses it to replay the two streams in the order they were written, rather than as two separate blocks. Only sessions recorded before this index existed fall back to the older order: stdout tail, then stderr tail.

Each console is capped in size (8 MiB per stream by default; STELLA_CONSOLE_CAP_BYTES changes it). A run that produces more output than the cap keeps the first eighth (how the run started) and the newest bytes, dropping the middle. A marker on stderr announces when this happens; it never gets mixed into the stream itself. A run that is killed outright (SIGKILL, power loss) can additionally lose up to one buffer's worth (about 64 KiB) of its very last output. That's the cost of enforcing the size cap while the run is still alive.

stop

Asks the run to stop the way Ctrl-C would: the signal reaches the whole process group (the run and every tool process it started), and the engine finishes the tool it's running before stopping at the next safe point, never mid-tool.

A run that hasn't stopped after 8 seconds is killed outright. Either way, the stop is recorded as cancelled, so a run you stopped by hand doesn't later look identical to one that crashed.

resume

Continues a killed run's interrupted turn from its last completed step. Every turn checkpoints at each completed step, the one point where the transcript is guaranteed to be complete, and every turn that ends normally (finishes, stops, or is cancelled) clears its checkpoint on the way out. So a resume point surviving means exactly one thing: the process died in the middle of a turn.

$ stella daemon list
ID                           STATUS       WHAT
ses-1785956826121-25167      Crashed ↩    stella: migrate the config loader to serde

↩ killed mid-turn with a resume point — stella daemon resume <id> continues it

$ stella daemon resume ses-1785956826121
▸ resuming ses-1785956826121-25167 — continuing its interrupted turn at step 14

The relaunch continues the same session (same id, same side files, the crashed attempt's console kept) as a fresh supervised process. Completed steps are already saved in the checkpointed transcript, tool results included, so nothing runs twice and no tool effect happens more than once. The overwrite-protection check isn't part of the checkpoint (it's kept only in memory, and the relaunch starts with none), so the resumed run re-reads any existing file in full before changing it. This also covers a file something else edited while the run was dead. A run that ended cleanly has nothing to resume, and resume says so instead of restarting it.

A turn that was running under an installed wrapper plugin re-enters that wrapper when possible. The wrapper saves its progress (task type, goal, plan position, test baseline) beside every checkpoint, and resume finishes the interrupted turn, runs the plan steps the crash never reached, then runs the check and verify steps on the finished work. A resume that ends with a verdict records its result as resumed_complete_verified and keeps the green checkmark.

$ stella daemon resume ses-1785956826121
  resuming ses-1785956826121-25167 at step 14 — completed steps stay done, nothing re-runs
  ! this was a wrapped run — resuming INTO it: the interrupted turn continues, then the witness and verify steps run on the completed work
    the pre-crash lint baseline is gone, so the lint-regression veto (#861) sits out this run
    an authored witness cannot be re-created after a crash — verification proceeds on the unauthored ladder

Two limits remain, and neither happens silently. A turn that was running in a candidate worktree resumes in the main workspace as a plain turn, because the candidate copy died with the process. And a turn saved by an older version of stella (or a run killed before it started working) has no progress record to restore, so it also resumes as a plain turn. In both cases, the run names every stage it can't restore (the verify command that won't re-run, the check that won't be credited), drops the green checkmark, and records its result as resumed_complete_unverified instead of resumed_complete. An unverified resume is still something you can act on; an unverified resume you were never told about is not. To re-verify, run stella run again over the same goal.

resume-all

resume continues one run you name. resume-all continues every run this machine interrupted, in one pass. This is the command a boot-time service runs, and it's also useful by hand after an unplanned reboot:

stella daemon resume-all --dry-run   # print the decision for every run, spend nothing
stella daemon resume-all

It prints one line per supervised run, continued or skipped, with the reason:

↩ continuing ses-1785956826121 — stella: finish the store migration
▸ skipping    ses-1785956700004 — ended deliberately — only an interrupted run is resumed at boot
▸ skipping    ses-1785956811930 — no resume point
  boot-time resume 1 of 3 for ses-1785956826121

What it continues

A supervised run that was interrupted: its liveness lock is gone, its workspace is still there, and it left a resume point behind. This is the same set of runs list marks Crashed ↩, plus a run that lived just long enough to record its own error before dying. Both are work that stopped without meaning to.

A run that finished, was stopped, was set aside, or stopped itself on purpose (a stuck loop that hit its limit, the step cap, an enforced budget, a scope review you cancelled) is never resumed. Each of those records a status on the way out saying the work ended rather than broke, and each one clears its resume point too. Work you ended on purpose is never resumed behind your back.

What it skips because nobody is there

A run that was waiting on a plan review when the machine went down is not resumed at boot. Resuming it wouldn't fail outright, but it would park again waiting for an answer, and since the sweep runs one run at a time to completion, it would sit there forever with every later run stuck behind it. So the sweep names it and moves on:

▸ skipping    ses-1785956700221 — waiting on an approval — answer it with `stella daemon attach <id>`, then `stella daemon resume <id>`

Waiting isn't the same as failing, so it doesn't use up one of the three attempts below. A run can wait across any number of reboots and still be resumable the moment you answer it.

What limits it

Each run gets at most three boot-time resumes. This count is saved to disk (~/.stella/services/resume-boot.json) and is written before the resume even starts, so a run that takes the machine down with it is still charged for the attempt. After three attempts, the run is left out of future sweeps and left for you to resume by hand with stella daemon resume <id>, since deciding to try again is meant to be a human choice at that point. Runs are resumed one at a time, since several turns resuming at once would mean several models spending money at once on a machine nobody is watching.

The attempt count limits how many boots a run can spend, but it doesn't stop one resumed turn from taking up the whole current boot forever (a stuck tool, a provider that never responds, a model call that retries endlessly). So each resumed run also gets a time limit, 30 minutes by default unless --ceiling MINUTES says otherwise. A run that hits this limit is never killed outright, since killing it mid-edit would be worse than the stall it's meant to fix. Instead it's stopped exactly the way stella daemon stop stops a run: asked to stop with SIGTERM, given time to reach a safe point and record its own ending, and only killed if it ignores that. Then the sweep reports it and moves to the next run:

⚠ ses-1785956826121 — did not finish within the 30-minute ceiling and was stopped at a safe boundary; the sweep continues with the runs behind it — `stella daemon list` shows where this one ended up

Both outcomes need no special handling. A run that stopped cleanly when asked ended the same as a Ctrl-C would, so the next sweep skips it and gives back its unused attempts; it was just slower than the time limit, and running stella daemon resume <id> by hand has no time limit. A run so stuck it had to be killed keeps its resume point and is swept again next boot, and the attempt it already used (a timed-out resume does count as one) is exactly what eventually retires it after three tries, since a run that hits the limit every boot is the kind of repeated failure the attempt limit exists to stop.

Everything resume does for a resumed turn applies here too, including the limits above: a turn interrupted mid-wrapper resumes as a plain turn and reports which steps it can't restore, written to the service console instead of a terminal.

install / uninstall

Supervision and resume both act on runs a person started. Neither one makes anything start on its own after a reboot, since nothing a process sets up can restart itself once it's gone. Starting again on boot is the operating system's job, and install is how you register one stella command you choose as a per-user service:

stella daemon install -- monitor --interval 300
stella daemon install --label night-fleet --keep-alive -- fleet watch
stella daemon uninstall night-fleet

On macOS, this writes a launchd agent (~/Library/LaunchAgents/sh.oxagen.stella.<label>.plist) and loads it with launchctl bootstrap. The command starts at every login, including the login after a reboot. On Linux, it writes a systemd --user unit (~/.config/systemd/user/stella-<label>.service), turns it on, and enables lingering (loginctl enable-linger) so the command starts at boot and keeps running through logout. If a launchctl or systemctl call fails, the service definition is still written, and the exact command to load it by hand is printed.

A registered command restarts from scratch each time, spending fresh money on a fresh turn every time the service manager launches it. So register standing tasks, such as a monitor, a fleet watch, or a run meant to keep going, and let one-shot runs stay simply supervised instead. Nothing is ever registered for you automatically.

To come back and continue the work a reboot interrupted, register the sweep instead:

stella daemon install --resume-all

This registers exactly one thing: stella daemon resume-all, under the label resume-boot, logging to ~/.stella/services/resume-boot.log. At every boot it continues the turns this machine killed, from their last completed step, and never restarts one from scratch. Its selection rule and three-attempt limit are described above. It takes no command of its own and refuses --keep-alive, since a sweep restarted every minute would use up its own attempt limit right away. stella daemon uninstall resume-boot removes it like any other service.

Two defaults keep a broken service from spending money unattended:

• A command that exits stays down until the next boot. --keep-alive turns on restart-on-exit, limited to once a minute, and on systemd also limited to five starts in ten minutes before the unit is marked failed.
• The program location is looked up each time it starts, and reported if missing. The service runs through a small shell wrapper that tries the path recorded at install time, then falls back to your PATH (plus the usual Homebrew and cargo locations), so an upgrade that moves the program doesn't break the service. If it still can't be found, one line naming what's missing is written to the service console and the service exits with code 127.

The service runs in the directory install was run from, and its console output goes to ~/.stella/services/<label>.log (both streams, in a directory only you can read). A service-run stella has no terminal, so it isn't supervised again (the service manager is its supervisor), and it doesn't show up in stella daemon list, which only lists supervised runs. Check its status using the operating system's own tools:

launchctl print gui/$(id -u)/sh.oxagen.stella.<label>   # macOS
systemctl --user status stella-<label>                  # Linux

uninstall unloads the service and deletes its definition; the commands above then report it gone. Uninstalling something already removed reports that and still succeeds. On Linux, lingering is left as you set it, since other user services may depend on it.

Where the state lives

Beside the session registry, under the user-level stella home directory (~/.stella/sessions/<id>/, STELLA_DATA_DIR overrides it): stdout.log and stderr.log hold the console output, console-index.jsonl is the write index that bounds and orders them, stdin is the prompt the supervisor saved for the run, supervisor.lock is the liveness lock, and approval-request.json / approval-answer.json appear only while a scope review is parked. All of these are readable only by you. The prompt is saved as a file rather than passed as an argument, so it isn't visible to every user on the machine through ps. The resume point itself isn't stored here. It lives in the workspace's own work journal, alongside the file changes it's a resume point for.