The Decision-Context Pack: What to Capture Before Something Breaks

Last week I laid out a 15-minute decision investigation: freeze one case, read what the system read, rebuild what was in effect, ask again with one thing changed, write the note. I ended it on a caveat. The method depends on artifacts you either kept or didn't, and when they're missing, the clock can't help you.

Here is what that looks like, as a composite of how it usually goes.

A complaint comes in about a booking the intake agent made three weeks ago. Someone opens the run to freeze the case and finds the raw request has already aged out. The ticket has a summary, written by a person, which is already an interpretation of the thing you're trying to investigate. At the rebuild step, the intake guidelines turn out to have been edited twice since then, in place, with no history. So the rebuild becomes a thread in the team channel asking whether anyone remembers what the page said at the end of the month.

Nobody did anything wrong. The investigation ran out of material before it ran out of time.

A decision-context pack is the minimum evidence you keep so that one consequential decision can be reconstructed later: what it saw, what was in effect, what changed, who owns the outcome, and what the team already learned from similar cases.

In July I wrote a decision-context checklist about what a single decision knows at the moment it's made: how it read the input, what else was live, how close the call was. That's context inside the decision.

This piece is about the context around it. Much of what an investigation needs doesn't live in the run at all. It lives in documents people edit without a deploy, in what normal looked like last month, in the record of who owns the outcome and what "wrong" means to them. None of that is in a trace, and all of it is gone or rewritten by the time someone asks.

An agent can tell you why it did something. The pack is what lets you check that answer against evidence the agent had no hand in producing.

The practical way to decide what goes in it is to take each step of the investigation and ask one question: what would I need to have kept, before anyone knew this case mattered, for this step to take minutes instead of an afternoon?

Logs record what the run did. The pack records what the run was standing on.

To freeze one case, you need to be able to find it later. That means an identifier that travels from the run to wherever a complaint will eventually arrive: the support ticket, the refund record, the customer's email thread. If the only way to find the run is by timestamp and a guess, freezing the case becomes the investigation.

It also means keeping the raw input longer than it takes a problem to surface. Retention windows are often set by storage defaults. The one that matters here is set by your slowest complaint path. If chargebacks, callbacks, or audit questions take weeks to arrive, a raw input that expires in days is gone before anyone knows to look for it.

To read what the system read, you need the input as received, not as summarized. Keep the request exactly as the agent saw it, plus whatever was retrieved and handed to the model on that specific run. Retrieval is easy to lose, because it's often computed fresh and not kept alongside the run. Re-running the retrieval today tells you what the system would read now, which is a different question.

To rebuild what was in effect, you need versions, including the ones that don't live in the repo. This is the step that stalled in the example above, and it's where the pack earns its keep.

Make a list of everything the decision reads. The prompt, the tool descriptions, the model version, the retrieval sources, and then the part that's easy to skip: policy pages, intake guidelines, canned instructions, knowledge-base entries, anything a person can change without shipping code. For each one, ask whether you could retrieve it as it stood on a given date. For the things in version control, yes. For the document someone edits in a browser tab, usually not.

What the investigation needs from those documents is their history. Snapshot them on edit, or on a schedule, into somewhere with timestamps.

Then keep one timeline that puts every change in order: prompt edits, model version changes (including provider-side changes behind an alias you didn't touch, where the provider exposes them), tool updates, and document edits side by side. "Did anything change?" should be answered by reading a list, not by asking around.

To ask again with one thing changed, you need a replay path that can pin versions. Somewhere outside production where you can re-run a decision with the prompt, the documents, and the model as they were on that date, then change exactly one of them. If the pack has the versions but nothing can load them, the step still falls back to guessing. Build the replay path before you need it, and test it on a case you already understand.

To write the note, you need to know who reads it and what normal looked like. The note goes to someone who owns the outcome, so the pack should name them, along with what a bad outcome means in their words ("an urgent request was booked as standard"), not the system's. And it should carry a rough sense of normal for the decision: a handful of reference cases with known-good outcomes, and how often each kind of outcome usually happens. You aren't building a baseline model. You're giving the investigator enough to say whether this case is unusual or just unlucky.

Last, keep the notes themselves. Every investigation you've already run is the start of a known-failures file. When the same shape comes back, the second investigation should start from the first note, not from the raw runs.

Keeping everything. The pack is scoped to the decisions that reach a consequence: the booking, the refund, the approval. Everything else is plumbing. A pack that captures the whole system is a data lake with a new name, and nobody will open it under time pressure.

Keeping raw input without thinking about what's in it. Longer retention on customer requests is longer retention on customer data. Decide what the investigation actually needs, redact the rest, and write down why you're keeping it. "In case we need it" isn't a reason anyone can audit.

Assuming the pack stays current. Workflows grow. The agent that handled standard bookings gets handed emergencies, then commercial accounts, and each new case type brings new documents it reads and new people who own the outcome. As I wrote on Tuesday, scope creeps faster than anyone's attention. The pack needs an owner, and a trigger: when the workflow takes on a new kind of case, the pack gets reviewed.

Treating the pack as an explanation. The pack makes an explanation checkable. Having every version of every document still leaves the reading, the replay, and the judgment to a person.

Take the last investigation note your team wrote, or the case you ran through the 15-minute method last week.

Go through it step by step, and sort each claim in the note into one of two columns: backed by an artifact you still had, or backed by someone's memory of how things were.

The memory column is your first pack. Each line in it is something that was true when the decision was made and survived only because a person happened to remember it.

You're not auditing your logging. You're finding out how much of your last explanation rested on recall, which is the part that won't be there next time.

Doing this by hand, for one workflow, is the exercise. Having the pack assembled continuously as the system runs, so the evidence already exists before anyone knows which decision they'll need it for, is the harder version, and right now teams keep hand-building it from scratch. It's the direction I'm building Nalyqor toward.