"Why Did It Do That?": A 15-Minute Decision Investigation

On Tuesday I wrote about a day nobody could explain: an intake agent books an emergency plumbing job into a standard slot next week, every dashboard stays green, and the day ends with a note to the owner that says "we think a guideline change caused it."

That note is the real deliverable of most AI investigations. Not a fix. A few sentences written to someone who has to decide whether to keep running the thing.

So this is about the fifteen minutes before that note gets written.

The instinct with a strange decision is to look until it makes sense. That search has no natural end, and it comes out of committed work, which is most of what makes diagnosis expensive in the first place.

A time-boxed investigation is doing something narrower. It separates the explanations you can show from the ones only a rerun would settle, and it does that on one case rather than on the pattern.

Fifteen minutes doesn't buy you the cause. It buys a note you can defend, and a short list of what would settle the rest.

Five blocks, and a stopwatch you actually obey.

Pick a single run. One request, one decision, one timestamp. Not "the agent's been off this week."

Write the complaint in the business's own words: an urgent request was booked as standard.

Then pin what you'll need before it moves: the input exactly as the system received it, the run itself, and the state of everything the agent reads.

An investigation that starts as a category can't be closed, because nothing you find can be checked against anything. One frozen case gives every later step something to be true or false about.

Read the input as the agent received it, not as a person summarized it in the ticket. Summaries are already interpretations, and you're investigating an interpretation.

Find the span that could support the decision it made. In Tuesday's case there's an obvious candidate, one clause: "we've shut the water off so it's not flooding anymore."

Write the competing reading next to it. Problem contained, or house with no water.

You're not asking whether the decision was right. You're asking whether the outcome is plausibly grounded in the input at all.

That fork decides the rest of your fifteen minutes. If a plausible reading supports the outcome, interpretation stays high on the suspect list, though the prompt, the retrieved context, or a policy line could still be what pushed the reading that way. If nothing in the input supports it, something else becomes the likelier driver, and your suspects change.

The developer's first question is "did anything change?", and the honest answer usually begins with "not in the code."

So list everything that decision reads, and check each one's state at the time of the run rather than today: the prompt, the tools and their descriptions, the model and its version, the context retrieved on that run, and the documents a person can edit without a deploy. Policy pages, intake guidelines, canned instructions, knowledge-base entries.

Most teams have change control over the first few and none over the last.

Anything a decision reads is part of the system, whether or not it lives in the repository. That's why "nothing changed" and "the behavior changed" are so often both true, and it's usually the block where a fifteen-minute investigation earns its keep.

If you have a way to replay that decision, re-ask it in a reproducible evaluation environment, never against production. A booking that actually books is not a sample.

Run it unchanged a few times first, to see whether the outcome is even stable on its own. Then change exactly one thing: drop the guideline line you added two weeks ago, or put the older version back.

If the outcome flips, you have evidence that the decision turned on that one change, rather than a story. If it doesn't, your leading candidate just got demoted, which is worth the same three minutes.

A rerun can't tell you the decision was correct, only whether it depended on the thing you changed. That distinction is the whole reason this block is short, and I've written before about what reruns do and don't establish.

No replay path yet? Stop at the candidate and name the test you would run if you had one. That still makes for a better note than the one Tuesday produced.

Three parts, in this order: what the system did, what you can show, and what's still unconfirmed along with what would settle it.

Tuesday's version was "we think a guideline change caused it." Here's the same belief after fifteen minutes:

> The request was read as already contained, and the line we added to the intake guidelines two weeks ago is the likeliest reason. Removing that line flips this booking to same-day in a replay. What we haven't checked is how many other requests it affected.

Same uncertainty. Different note. The owner can act on the second one, because the uncertainty is attached to a specific claim instead of sitting in front of the whole sentence.

It won't tell you how often this has happened. That's a question about a population of decisions, and it needs them captured the same way over time, which no investigation can retrofit on a Thursday morning.

It produces one candidate with evidence attached, and a named way to test it. That is less than a cause.

It depends on artifacts you either kept or didn't: the input as received, the versions in effect at that moment, the edit history of the documents the agent reads. Where those are missing, the clock can't help you, and that absence is the finding. It's also next week's piece.

And rolling the change back is not an explanation. A rollback ends the symptom and closes the investigation in the same motion, which is why the second occurrence starts from the runs again, at full price.

Run it on a case you already closed and believe you understand.

You already know the answer on that one, which is the point. You're checking whether the note the method produces matches the explanation your team accepted. Where the two differ, it's usually the "what you can show" line coming up thinner than memory had it.

Give it fifteen minutes and stop. If you're still going at thirty, either the case wasn't frozen tightly enough or an artifact you needed wasn't kept, and both of those are worth knowing before the next one.

Doing this by hand, on one case, is the exercise. Having the timeline already assembled when the owner asks, so the note starts from evidence instead of memory, is the direction I'm building Nalyqor toward.