Your refund-approval agent processes a few thousand requests a day. Almost all of them are unremarkable: a return within policy, a straightforward approve. You can't review all of them, and you shouldn't try to. The ones worth worrying about aren't the ones that look wrong. Nothing about an approval looks wrong at the moment it's made. They're the ones that were close.
Ask which of yesterday's approvals were mistakes and you won't get an answer, not because nobody's looking, but because "wrong" isn't visible from the outside until it shows up later as a chargeback pattern or a support escalation. What you can ask today, before any of that surfaces, is which of yesterday's approvals behaved like close calls. That question has an answer, and you don't need a full decision-record pipeline in place to start getting it.
Close calls, not wrong calls
I wrote about decision margin a while back: how close a call was, distinct from which way it went. A decision that barely won is a different animal from one that won decisively, and most systems can't tell you which kind you're looking at after the fact.
Margin matters more now than it did then, because of a constraint I wrote about more recently: actually explaining why a specific decision landed where it did is expensive, and it usually falls to one of a small number of people who happen to hold the system's history in their head. You cannot spend that person's time on every approval. You can spend it on the ones that were unstable.
One definition before the framework, because the word is doing real work here. What follows measures stability, not the system's self-reported confidence: whether a decision holds up when you ask again under slightly different conditions. That is not the same thing as correct, and it is worth being precise about the difference now rather than running into it in the guardrails section.
The reframe is the whole piece: stop asking which decisions were wrong. Start asking which decisions were unstable enough that they could easily have gone the other way. The first question needs an outcome you don't have yet. The second you can answer this afternoon.
Three tiers, and what each earns
You don't need a probability score baked into the system to sort decisions this way. You need to know, for a given decision, how often it would land the same way if you asked again under conditions that were close but not identical.
Signal | What it looks like | What it earns |
|---|---|---|
Stable | Repeated runs stay consistent | Lower triage priority; spot-check periodically |
Sensitive | Small changes sometimes flip the outcome | Add to this week's review queue |
Unstable | Small changes repeatedly flip the outcome | Prioritize for immediate review |
To get the signal, take a sample of yesterday's decisions at the point closest to consequence, the last step before something actually happens, a refund issued, a message sent. First, rerun each one several times under the exact same production configuration. That tells you how stable the decision is under its own stochasticity alone, before you've changed anything about it.
If a decision is consequential enough to matter and survives that first pass, introduce one controlled, meaning-preserving change at a time: reorder a piece of context, rephrase a field without changing what it says. That second pass is asking a different question than the first, whether the judgment is sensitive to presentation rather than substance, and blurring the two together muddies what the resulting signal actually means. Change one thing per rerun. Know which question you're answering.
This works as a proxy for the reason you'd expect: a decision that survives repeated runs and small presentation changes looks locally stable. One that flips is telling you the outcome depended on something that could easily have gone differently, which is a fair definition of unstable whether or not you ever learn what the "right" answer was.
If your system already exposes something like a confidence or ranking score, that's a cheaper first filter, and it's fine to use it that way. Just don't stop there for anything consequential. A score is the system's own account of its certainty, and a system can be confidently, consistently wrong. Reruns tell you something a self-reported score can't: whether the decision is actually stable under variation, not just whether the system claims to be sure.
What this doesn't tell you
Stability can be a useful proxy for whether a call was fragile, not for whether it was correct. A system running on a stale assumption, one signal that used to matter and quietly stopped being retrieved, can produce the same wrong answer every single time, with total consistency. That decision will sort into your "stable, lower priority" tier, and it will sit there, comfortably wrong, until something else surfaces it. Triage by stability catches fragility. It does not catch a stable system that has been wrong the whole time in exactly the same way.
This is also why the framework doesn't answer "which decisions are safe." It answers "which decisions deserve scarce attention first." Instability is a reason to look. Stability is a reason to deprioritize, not proof that the decision was right.
Run the reruns against a reproducible evaluation setup, never against live production. A refund that actually issues five times because you were sampling it is not a sample, it's an incident.
And don't let this quietly recreate the bottleneck it's meant to relieve. If every unstable decision gets routed to the same one person who can explain anything, you've built a faster path to the same jam. The point of triage is to shrink what reaches that person to the fraction that actually needed them, not to guarantee they still see everything that might.
Start with one decision point, one day
Pick the decision closest to consequence, the same one you'd start with if you were capturing decision context for the first time. For one day's worth of real decisions at that point, rerun each several times under the same captured configuration. For the consequential ones that stay stable, introduce one meaning-preserving perturbation at a time and note whether the outcome changes.
Sort the results. The unstable ones are this week's review queue. Not the ones that already look stable. Give it a week before you decide whether to build anything more permanent on top of it.
Doing this by hand, for one decision point, for a day, is the whole exercise this week. Doing it continuously, across every decision point a system makes, at whatever volume production actually hands you, so that the one person who can explain an unstable decision primarily gets paged for the decisions that actually showed instability, is the harder version. That's the direction I'm building Nalyqor toward. The goal was never fewer decisions to explain. It's making sure the ones that reach that person are the ones that earned it.