Drop stella in as your agent engine

You shipped an AI feature. The first version was a while loop around a completions call, and it worked. Then it went into production, and the loop grew: a step cap, because a model called the same tool nine times. A way to shorten the history, because it outgrew the context window. A retry, then a smarter retry, because a 429 and a 400 shouldn't be treated the same way. At some point it stopped being a feature and started being a system nobody really owns.

None of that is your product. All of it sits on your critical path.

This guide replaces that loop with stella's, without giving up anything you care about — your models, your keys, your tools, your data, your bill.

The move, in one paragraph

You run stella-serve as a process next to your app. When a turn starts, the engine drives the loop and asks your app to do the two things that touch anything sensitive: call the model, and run the tool. The engine has no HTTP client, no TLS stack, and no provider code of its own, so it simply can't make either call itself.

That's the whole design, and it's why this is safe to put inside a product: the engine is just a scheduler for work your app does. It never holds a credential, so it can never leak one.

Before you start

Step 1: Get a server running

Two commands. There's no stella serve subcommand, and no release download includes the server — it's a separate program with its own binary.

cargo build --release -p stella-serve --bin stella-serve

STELLA_SERVE_BIND=127.0.0.1:8137 \
STELLA_SERVE_TOKEN="$(openssl rand -base64 32)" \
./target/release/stella-serve
# stella-serve listening on 127.0.0.1:8137

It's a fast build — the server only pulls in two libraries, not the full CLI or the terminal UI.

Write down the version it prints, next to your client code. The message format only ever adds fields, but pinning a version and running a round-trip test is how you find out about a change on your own schedule instead of in production.

Step 2: Port the loop, not the feature

Here's the part people over-plan. Your existing feature already has the two functions the engine needs. Find them:

Everything between those two functions — the step counting, threading the history, shortening it, retrying — is what you get to delete.

The complete working host is about eighty lines of plain Node with no dependencies. The reference page walks through the six routes, the four message types, and the error types. Don't re-read all of that here — open it in another tab and come back once your first turn completes.

Step 3: The payoff — a test that costs nothing

This is the real reason to do this port, and it's worth doing on day one instead of day thirty.

Because your app owns the model call, you can swap in a scripted reply instead of calling your real gateway. The engine can't tell the difference: it sends the same messages, runs the same steps, threads the same history, and reaches the same outcome.

So a full, multi-step agent turn becomes an ordinary unit test. No network, no key, no spend, and it runs in milliseconds.

import { test } from "node:test";
import assert from "node:assert/strict";

test("the model can call a tool and answer from its result", async () => {
  const replies = [
    { tool_calls: [{ call_id: "c1", name: "get_weather", input: { city: "Paris" } }],
      finish_reason: "tool_calls" },
    { text: "It is 18C and clear in Paris.", finish_reason: "stop" },
  ];
  let n = 0;

  const outcome = await runTurn({
    prompt: "What is the weather in Paris?",
    // your gateway, replaced by an array
    onProviderRequest: () => withUsage(replies[n++]),
    onToolRequest: (name, input) => dispatch(name, input),
  });

  assert.equal(outcome.status, "completed");
  assert.match(outcome.text, /18C/);
  assert.equal(n, 2, "one call to ask for the tool, one to answer from it");
});

That test covers more than it looks like. The engine recorded the assistant's tool call, matched your result to it by call_id, rebuilt the conversation in your provider's format, and ran the second step from there. That's the machinery you deleted in step 2 — now under test, for free.

Write tests like this for the cases you can't afford to hit in production: a tool that errors, a model that returns bad arguments, a rate limit in the middle of a turn, a turn that hits its step cap. Each one is just a scripted array, and none of them cost a cent.

Step 4: Before it carries traffic

A short list. Every item here is a real failure someone has actually hit.

Some things genuinely don't exist yet: no resuming a stream, no server-side conversation state, no approval routes, no /metrics endpoint. Plan around these instead of discovering them the hard way: not built yet.

Where this goes next

You now have an agent loop that runs in CI without needing a key. The natural next step is to make CI actually care about it, because the engine is a real dependency now, and dependencies can break.

Related reading