Drop Stella in as your agent engine

You shipped an LLM feature. The first version was a while loop around a completions call, and it worked. Then it met production, and the loop grew: a step cap, because a model called the same tool nine times. A truncation, because the history outgrew the context window. A retry, then a smarter retry, because a 429 and a 400 should not be treated the same. Somewhere in there it stopped being a feature and started being a distributed system nobody owns.

None of that is your product. All of it is on your critical path.

This guide replaces that loop with Stella's, without giving up a single thing you care about — your models, your keys, your tools, your data, your bill.

The move, in one paragraph

You run stella-serve as a process next to your app. When you start a turn, the engine drives the loop and asks your app to do the two things that touch anything sensitive: make the model call, and run the tool. It has no HTTP client, no TLS stack, and no provider adapter, so it is not able to make either call itself.

That inversion is the whole design, and it is why this is a reasonable thing to put inside a product: the engine is a scheduler for work your app performs. It never holds a credential, so it cannot leak one.

Before you start

Step 1: Get a server running

Two commands. There is no stella serve subcommand and no release tarball ships the server — it is a separate crate with its own binary.

cargo build --release -p stella-serve --bin stella-serve

STELLA_SERVE_BIND=127.0.0.1:8137 \
STELLA_SERVE_TOKEN="$(openssl rand -base64 32)" \
./target/release/stella-serve
# stella-serve listening on 127.0.0.1:8137

It is a fast build; the server pulls in two crates, not the CLI or the TUI.

Record the version it prints next to your client code. The wire format is additive, but a pin plus a round-trip test is how you find out about a change on your schedule instead of in production.

Step 2: Port the loop, not the feature

Here is the part people over-plan. Your existing feature already contains the two functions the engine needs. Find them:

Everything between those two functions — the stepping, the history threading, the compaction, the retries — is what you are deleting.

The complete working host is about eighty lines of dependency-free Node, and the reference page walks the six routes, the four frame types, and the error classes. Do not restate it here; open it in another tab and come back when your first turn completes.

Step 3: The payoff — a test that costs nothing

This is the reason to do the port, and it is worth doing on day one rather than day thirty.

Because your app is the model, you can substitute a scripted reply for your gateway. The engine cannot tell the difference: it emits the same frames, runs the same steps, threads the same history, and settles the same outcome.

So a full multi-step agent turn becomes an ordinary unit test. No network, no key, no spend, milliseconds.

import { test } from "node:test";
import assert from "node:assert/strict";

test("the model can call a tool and answer from its result", async () => {
  const replies = [
    { tool_calls: [{ call_id: "c1", name: "get_weather", input: { city: "Paris" } }],
      finish_reason: "tool_calls" },
    { text: "It is 18C and clear in Paris.", finish_reason: "stop" },
  ];
  let n = 0;

  const outcome = await runTurn({
    prompt: "What is the weather in Paris?",
    // your gateway, replaced by an array
    onProviderRequest: () => withUsage(replies[n++]),
    onToolRequest: (name, input) => dispatch(name, input),
  });

  assert.equal(outcome.status, "completed");
  assert.match(outcome.text, /18C/);
  assert.equal(n, 2, "one call to ask for the tool, one to answer from it");
});

What that test actually covers is larger than it looks: the engine recorded the assistant's tool call, matched your result to it by call_id, rebuilt the conversation in your provider's shape, and ran the second step from it. That is the machinery you deleted in step 2, under test, for free.

Write these for the cases you cannot afford to hit in production — a tool that errors, a model that returns malformed arguments, a rate limit mid-turn, a turn that hits its step cap. Each one is a scripted array, and none of them cost a cent.

Step 4: Before it carries traffic

Short list, each item a real failure someone has had:

There is also a list of things that genuinely do not exist yet — no stream resumption, no server-side conversation state, no approval routes, no /metrics. Design around them rather than discovering them: what is not built yet.

Where this goes next

You now have an agent loop that runs in CI without a key. The natural next move is to make CI care about it — because the engine is a dependency now, and dependencies regress.

Related reading