You already know the moment. A symptom shows up somewhere downstream: a service that starts timing out, a customer complaint that doesn't match anything in the trace, a number that finally moves. You start where the pain is, and you work backward, and eventually you find a step that looks wrong.
Then the harder question shows up. Is that the failure, or just the place it happened to surface?
Knowing where you ended up isn't the same as knowing where you started. Photo: Denise Jans on Unsplash
I mapped this out recently in The Anatomy of a Silent Failure: a silent failure isn't a single bad decision; it's a five-stage sequence, and most of that sequence happens while every instrument you have reports healthy. Origination, where a decision first diverges from what was actually warranted. Local coherence, where that decision looks completely fine when examined alone. Absorption, where your logs, traces, and evals each grade a different artifact than the one that actually broke. Compounding, where the divergence gets treated as precedent, and its footprint grows. Surfacing, where something finally crosses a threshold, usually somewhere structurally distant from where it started.
That's the framework. This piece is the field guide. For an incident sitting in front of you right now, here's how to figure out which of the five stages you're actually looking at, one stage at a time, using an investigation you can run this week.
How to tell which stage you're in
Same pattern, five times. Ask one question, look for three things, and check the sentence that tells you whether you've actually found it.
Origination
Ask
Is there an unlabeled precedent upstream?
Look for
A borderline decision treated as routine
An exception that became the new normal
The earliest instance of this decision type in your history, not the incident itself
You're probably here if...
You find one reasonable-looking decision that quietly expanded the rule, and nothing in the record marks it as an exception.
### Local coherence
Ask
Would each individual decision pass review on its own?
Look for
- Five to ten decisions of the same type, reviewed one at a time, not as a batch
- Each one judged in isolation, the way the system's own checks judged it
- No single decision that looks obviously wrong
You're probably here if...
Every decision looks fine alone. That's not a dead end. It's confirmation: the failure lives in the sequence, not the step.
Absorption
Ask
What does each instrument actually grade?
Look for
Your logs, trace, evals, and alerts, checked one at a time
Whether each one grades output, execution path, or interpretation
Instruments that report healthy because they were never pointed at interpretation in the first place
Every single print looks fine on its own. The line is where the story actually is. Photo: Caleb Jack on Unsplash
You're probably here if...
Everything reports healthy, and none of your instruments were ever built to see the thing that broke. That's not a monitoring gap you patch with another alert. It's the wrong layer being watched.
Compounding
Ask
Has "acceptable" quietly widened?
Look for
The earliest instance of this decision type, compared against a recent one
Aggregate metrics that stayed flat or quietly improved over that window
Fewer flags or fewer reviews that look like progress but aren't
You're probably here if...
The category has visibly widened, and nothing in your metrics flagged it, because a slow directional shift doesn't cross a threshold. It just moves one.
Surfacing
Ask
Is this the failure, or the first place it became visible?
Look for
The fix you're about to ship, stated in one sentence
Whether that sentence names a rule, or a specific package, maintainer, or customer
A narrower fix that would prevent recurrence under a different name
You're probably here if...
You can't find a narrower fix that would hold. Whatever you're about to change applies to the actual interpretation that drifted, not just the instance that happened to trip the wire.
The field guide at a glance
| Stage | Ask | Check | Confirming signal |
|-----------------|--------------------------------------------------------------------------------------------|----------------------------------------------------------------------|-------------------------------------------------------|
| Origination | Did a borderline case succeed once, outside policy, without being flagged as an exception? | The earliest instance in your history, not the incident itself | One reasonable decision, never marked as an exception |
| Local coherence | Would each decision in the chain pass review alone? | Five to ten decisions, reviewed one at a time, not as a batch | Every one looks fine alone |
| Absorption | What does each instrument actually grade: output, path, or interpretation? | Every instrument on your list, one at a time | None of them were built to grade interpretation |
| Compounding | Would today's version of this decision get made the same way on day one? | First instance vs. a recent one, plus metrics for the window between | The category widened and no metric flagged it |
| Surfacing | Would a narrower fix actually stop this from recurring under a different name? | The fix you're about to ship, stated in one sentence | No narrower fix would hold |What this doesn't give you
A few honest limits, because a checklist that doesn't admit its edges gets trusted more than it should.
This assumes you already know roughly where the symptom is. It won't tell you which of a dozen decision points to start pulling on if you have no lead at all; that's still judgment, not a formula.
The Local coherence check only proves the individual decisions were reasonable alone. It doesn't, by itself, prove which one started the pattern. The earliest-instance comparison under Compounding is what narrows that down.
These questions narrow the search. They don't produce a verdict. A fix can pass every check on this list and still be wrong in a way nothing here catches, so treat a clean run through the guide as evidence, not proof.
And if your system doesn't record individual decisions at all, only final outputs, most of this guide can only run going forward, once you start capturing decision context, not backward on an incident that already happened. I wrote about how to do that cheaply, without rebuilding your stack, in What to Log When Logs Aren't Enough.
Where to start this week
Pick one incident you already called fixed, ideally one where the fix was a narrow rule: exclude this package, flag this customer, block this input. Run just the last two questions against it, Compounding and Surfacing. Would today's version of that decision still get made the same way somewhere else? Would a narrower fix have prevented recurrence under a different name?
That's the cheapest version of this guide: two questions, one incident you already have the history for, no new instrumentation required.
Doing this by hand, on one incident, is the whole exercise this week. Running it continuously, across every decision, so Origination gets caught before it's had five weeks to compound into a Surfacing event, is the harder problem. That's the direction I'm building Nalyqor toward.
But you don't need it to run the version above. Pick the incident, ask the last two questions, and see how far back the actual failure really goes.