The Solo-Team AI Reliability Stack: What to Instrument First post

Go looking for advice on instrumenting an AI agent and you'll find lists written for a platform team. Capture every span. Log every prompt and completion. Version your embeddings, wire up an eval suite, stand up a trace viewer, tag everything. All of it defensible, none of it prioritized, and every list quietly assumes there's someone whose job is to look at what gets captured.

On a five-person team, there isn't. Last week I wrote about the judgment job that lands on small teams when they put agents into production without an ML engineer: someone has to say whether a given decision was sound, and it defaults to the workflow owner at the worst possible time. This piece is the practical half of that argument. If that job is yours, and you get an afternoon rather than a quarter, what do you actually instrument, and in what order?

The order matters more than the list. Get it backwards, and you'll do real work, capture real data, and still learn about your first silent failure from a customer.

The usual instinct is to instrument outward from the model: first the calls, then the traces, then dashboards on top. That order is inherited from infrastructure monitoring, where it works, because infrastructure fails loudly. When the database is down, the instrumentation's job is diagnosis, and the closer to the machinery you are, the better.

Agents invert this. The failures that matter don't throw errors. A refund agent that starts approving what it used to escalate produces clean traces. So instrumentation built from the model outward answers "what did it do?" in great detail while the question you actually have, "did something go wrong?", has no signal at all. I've written a field guide to diagnosing these failures after the fact; the point of instrumenting is to stop needing it so often.

So build in the other direction. Start where the agent's decisions touch the world, and work backward toward the model only as real questions pull you there.

A trace shows what the agent did. A disposition shows whether anyone had to undo it.

That second signal is the one a solo team can't do without, and it's rarely the signal standard instrumentation lists put first.

Five layers. Each one answers a question, and each layer is worth building only when the layer before it keeps raising that question. That pull rule is the whole discipline: on a small team, instrumentation should be dragged into existence by questions you actually faced, not installed speculatively because a list said so.

For one workflow, record what became of each run's output downstream: accepted, corrected, escalated, reversed, quietly redone by a person. Not whether the run succeeded technically. What the world did with the result.

This is the smoke detector layer. It doesn't watch the fire; it notices smoke, and that's why it catches the failures that traces can't. A bad decision that throws no error still leaves a downstream trail: the ticket gets reopened, the refund gets clawed back, a person edits the draft heavily before sending. Those events already happen in systems you already run. The instrumentation is mostly a join: making sure each of them can be tied back to the run that caused it.

You are not building analytics here. A weekly count of dispositions per workflow, even kept by hand at first, is enough to make a drift visible as a trend instead of an anecdote.

When a disposition goes bad, the next question is what the agent actually did. So the second layer is a run record you can reach from the first: input, the steps taken, the output, findable from the ticket or the refund or the PR in one hop.

Most teams partially have this already. Provider dashboards keep calls, frameworks emit traces, your app logs the request. The gap is rarely capture. It's retention and reachability: the trace exists but expires before the question arrives, or it exists but nothing connects it to the complaint, so every investigation opens with a hunt for the right run. Fix the join before you add any capture. A modest record you can reach in one step beats a complete one you have to go dig for.

The test for this layer is concrete: pick last week's worst output and time how long it takes to get from the complaint to the full run behind it. If that's minutes, this layer is done, and adding more capture here is procrastination with a dashboard.

Layers one and two tell you something went wrong and what happened. The first time they leave you staring at a plausible-looking trace unable to say why the agent chose what it chose, you've hit the limit of what execution data holds, and you build the third layer: capturing the context around the decision itself, at the moment it's made.

I've published the full checklist for this: what kinds of context exist, the places worth watching, the rules that keep the practice honest. The solo-team version is that checklist applied with maximum restraint. One decision point only, the one closest to consequence: the last choice before something irreversible happens, the refund approved, the message sent, the write committed. How the input was read, what options were live, how close the call was.

One point sounds too small to matter. It isn't, because consequence is where all the downstream signal from layer one already points. When a disposition goes bad, the last consequential decision is usually the highest-leverage place to start, and having context there converts "the trace looks fine" into an actual answer more often than instrumenting five upstream points would.

In the companion piece I argued that judging an agent's decision depends on history rather than skill: knowing that this agent always retries the CRM lookup twice and that's fine, but three times means the integration is degrading. On teams without an ML engineer, that history lives in the workflow owner's head, which means it evaporates under pressure and walks out the door with them.

The fourth layer is embarrassingly low-tech: write it down. A short, living note per agent workflow. Typical tool sequence on the main task. Usual retry behavior. Roughly how often it escalates, and what a normal week's dispositions look like from layer one. What "weird but fine" has looked like before, so the next investigator doesn't rediscover it at 11pm.

This layer converts every investigation you do into cheaper future investigations, which is the compounding the other layers can't give you. It also goes stale, which is its known failure mode; more on that below.

Last: divide the model bill for the workflow by the finished work it produced, monthly. Cost per resolved ticket, per merged review, per processed claim. One number, tracked over time.

This layer isn't about savings, and I'd build it last precisely because cost work has its own pieces. Its job in the reliability stack is to be a tripwire: behavior drift often shows up as spend drift before anyone reads a transcript. Retry loops, growing context, a routing choice that quietly changed. A rising cost per outcome with a flat disposition trend says go look at the run records now, while the only casualty is money.

Dispositions lag. A bad decision can take days to surface as a reversal, and some never do (the customer just leaves). Layer one is a detector with latency, not a guarantee, which is why the layers behind it exist at all.

People erode the signal quietly. The way layer one most often fails is a teammate who stops delegating to the agent without telling anyone, so the disposition trend looks stable because the risky work left the workflow. The fix costs one standup question: what do you now do yourself that you used to let it handle?

The pull rule cuts both ways. Building layers three through five before anything pulled you there is how a two-person team ends up maintaining instrumentation instead of a product. But the rule also means when a layer does get pulled, you build it then, not two incidents later.

Written normal rots. A baseline note nobody has touched in months describes a previous agent. Date the entries, and treat an investigation that contradicts the note as a prompt to update it, not evidence that the note was a bad idea.

Pick the workflow whose output touches something irreversible, and build layer one for it. Where does its output land: a ticket, a PR, an approval queue? Add the disposition tag at that landing point, then count last month's by hand, once, to give yourself a starting trend. An afternoon, no new tools, and within a week you'll have your first real signal on whether the agent's decisions are quietly costing you undo work.

Then let the questions pull. When a bad disposition shows up and you can't reach the run, build the join. When you can reach the run and still can't explain it, instrument the decision point. Each layer earns the next.

What a solo team can't do by hand is run this continuously: every workflow, dispositions joined to runs, decision context on every consequential call, baselines that update themselves instead of rotting in a note. That's the always-on version of the job, and it's what I'm building Nalyqor toward, on the belief that the production judgment small teams can't hire should be something they can switch on. You don't need it for this week's move. You need one workflow, one tag, and the willingness to find out what's been getting undone without you noticing.