Your Agent's Decisions Should Be Records, Not Logs

Ask a simple question about your agent. How many refusals did it issue last month, and how many were refusals the policy allowed?

If the agent's decisions only exist in logs, answering that usually means reconstructing the decisions first. You can search for a refusal, read a few, and form an impression. What you don't already have is a consistent population of decisions you can count and compare, and so you can't easily tell whether the number moved after something changed.

The information is in the logs. It's just stored for debugging one request at a time, which is a different job from keeping an account of decisions.

Every agent decision has the same basic shape, whatever the agent does. Something was decided, about something, under some rules, on some evidence, with some result.

A decision record is one entry per decision, shaped the same way every time so that decisions can be compared with each other. Written as structure rather than buried in a transcript, the decision becomes something you can look at across many runs.

A usable record has to preserve enough to tell:

- what kind of decision happened,

- what outcome resulted,

- what state the system was in when it decided,

- whether enough evidence survived to examine it,

- and what someone learned if they later investigated it.

The evidence property matters most. A missing trace is easy to read as a good decision, because nothing looks wrong. A record that says the evidence was incomplete stops that from happening.

The reasoning and the full inputs are still worth keeping. The principle is that the comparable record should not need to contain every raw input or transcript. The record is for seeing patterns, and it stays small enough that a person can read it.

Mixing the two is how a debugging artifact turns into a privacy problem. Anyone who can read the record can then read everything in it.

A record only helps if the people who need it can reach it. For most companies that means the warehouse or lake the finance and operations teams already query, not a new tool the agent's team owns.

Sales is the familiar example. Nobody expects a salesperson to read a call transcript to know where a deal stands. The CRM keeps one record per opportunity, with the stage, the value and the date, and the business builds its forecasts on that. Nobody has to ask engineering for it.

Agent decisions can work the same way, as first-class operational data, like opportunities, invoices or tickets. One record per decision, in the same store as the rest of the business's operational data, means the questions get asked by the people who own the outcome, not only by whoever wrote the agent.

Once decisions are records, a few questions that were hard to answer become routine.

Is the refusal rate moving, and did it move when the instructions changed? Did a model update change which decisions the agent makes? How many decisions can't be judged at all, because the evidence wasn't in the run?

None of those needs a special tool. They need the decisions written down in one consistent place, which is the part most teams haven't done.

A dashboard shows the questions someone predicted when they built it. When a failure is new, there's usually no panel for it.

A record holds what happened, so whoever has the question can ask it. The problem nobody expected is often visible the first time someone looks at the decisions side by side.

A record is only as good as what went into it. If it says a decision was expected and nobody checked the evidence behind it, you have a confident label with nothing under it.

Two things to settle before the first record is written.

Who can read it. Decisions often include customer details. Remove what doesn't need to be there before the record is written, and keep the access rules the same as the systems the decisions came from.

How long it lives. Decide the retention period when you start, not when someone asks. A record that outlives the data it refers to is harder to defend than one that expires on a known schedule.

Pick one decision your agent makes often. Write down, on paper, what a record of that decision would need to say. Then ask the question you couldn't answer last month, and see how many of those properties you'd have needed to answer it.

If the answer is that you don't have them, that's the finding. It tells you what your logs were never keeping.

This is the direction I'm building Nalyqor in: each agent decision recorded as a structured record a team can look at across runs, and written into the store the team already uses. Some of that exists now. Some doesn't yet. I'll say which is which when it ships, because a record you can't trust is worse than none.