A few weeks back, I wrote about why decisions disappear the moment execution finishes; the logs show what ran, not what almost ran instead, and by the time something breaks, that information is gone for good. This piece isn't the argument. It's the checklist: what to hold onto, where to look for it, and the two rules that keep the practice honest once you start.
What's live in the moment doesn't stay long enough to look back at. Photo: Thomas Stephan on Unsplash
Five kinds of decision context worth preserving
Every decision your system makes carries the same five categories of information, and all five tend to exist somewhere in the system for a moment before they're discarded.
Interpretation — how the system read the input in the first place. This is usually already happening; it just isn't written down anywhere durable.
Alternatives considered — the full set of options that were actually live, not just the one that won.
Relative preference — how those alternatives ranked against each other, not only which one came out on top.
Decision margin — how close the call was. A decision that barely won is a different animal from one that won decisively, and right now most systems can't tell you which kind you're looking at after the fact.
Context in play — what was available to the system at the moment it decided, with enough provenance that you could later ask which parts actually mattered.
Most of this information already exists briefly inside the system. The gap isn't computation. It's that almost none of it survives the decision itself.
Where to look for it
You don't need to track this everywhere. It only matters at the handful of places where the system is actually choosing between things:
Intent or routing — where the system decides what kind of request this even is.
Tool selection — where it picks an action. Usually the highest-leverage point, because it's also the one that fails silently: right tool, wrong order, wrong turn.
Retrieval — where it ranks candidates and keeps a subset. The alternatives that didn't make the cut matter as much as the ones that did.
Plan steps — where a multi-step agent commits to the next move given everything before it. What drove step four is often something decided at step one.
The point closest to consequence — the last decision before a side effect actually happens: a write, a message sent, a refund approved. If you can only watch one place, watch this one.
Five kinds of place, not fifty. Anywhere else is plumbing, and instrumenting plumbing is how projects like this drown in their own data before they produce anything useful.
The shape of what you're preserving
You don't need a schema to start, just a mental model of what belongs together:
Decision Point
├── Interpretation — how the input got read
├── Alternatives Considered — what else was live
├── Relative Preference — how they ranked against each other
├── Decision Margin — how close the call was
└── Context In Play — what was available, with provenanceHowever you end up recording this, keep it attached to the specific decision it came from, not floating loose in a general log. A decision record that can't be traced back to the moment it happened is just another unstructured note.
The point isn't a schema. It's knowing which drawer a piece of context belongs in, and being able to find it again. Photo: Jaron Mobley on Unsplash
The two rules that make this trustworthy
Rule 1: always mark what you measured versus what you inferred. Some of this information is available in the moment; some of it (which piece of context actually mattered, how stable a decision is under slightly different conditions) can only be estimated afterward, from a sample. Both are useful. The failure mode is letting an estimate sit next to a measurement, indistinguishable, until someone downstream trusts a guess as if it were fact. Whatever you build, keep that distinction visible everywhere the data travels.
Rule 2: reconstruction belongs in a reproducible evaluation environment, never against production. If you're going to re-examine a decision under slightly different conditions, do it somewhere the side effects aren't real. This isn't just a safety rule; it's what makes the practice affordable at all. You can afford to be thorough on a sample precisely because nothing you do there has consequences.
The gut-check before you start
A few ways this kind of effort tends to go wrong, and the correction for each:
Trying to capture everything → Stick to the handful of decision points that actually matter. Nothing else.
Treating reconstruction as continuous → Reserve it for failures, close calls, and a small ongoing sample. Not everything, all the time.
Letting records pile up as unstructured notes → Keep them compact and traceable back to the decision, not buried in free text.
Assuming the measured-versus-inferred line is obvious → It's obvious to you today, while you're building it. It won't be obvious to whoever reads the postmortem next year.
Where to start this week
Pick one decision point, the one closest to consequence, and start with the first three: interpretation, alternatives considered, and decision margin. In most systems, all three already exist somewhere for a moment before they vanish. The work isn't inventing new computation. It's deciding, deliberately, what's worth keeping before it disappears.
Give it a week before you touch reconstruction at all. You'll likely find that what you kept answers most of the question you were opening old traces to ask.
Doing this by hand, on one decision point, is the whole exercise this week. Making it continuous, structured, and trustworthy across every decision a system makes, without every team hand-building the same infrastructure from scratch, is the harder problem; and it's the direction I'm building Nalyqor toward. But you don't need it to start noticing what you're currently letting disappear.