A team I was working with had a support-triage pilot that had been "almost ready" for four months. Every review went the same way: someone pulled up a handful of recent transcripts, they looked fine, and the meeting ended with "let's revisit next sprint." Nobody could point to a specific blocker. Nobody could say what "ready" would actually look like if they saw it.
I wrote recently about why this happens: a pilot proves the system can do the thing. Production asks a different question: whether anyone will know when it's wrong and be able to explain why. That piece was the diagnosis. This one is the checklist: four things you can actually check this week, using the pilot you already have, before you walk into that meeting again.
The pilot already built the path to the gate. Nobody's checked whether the gate actually opens. Photo: Aleksandar Manasiev on Unsplash
Same transcript, different question
A pilot review asks one question: does the output look right. Someone reads a handful of transcripts, nods, and the pilot inches forward. A production review asks a different question: when this is wrong, will anyone know, and can they explain what happened.
Those two questions can both get asked while looking at the exact same transcript, and they produce completely different answers. The transcript that convinces a room the pilot is ready is often silent about the thing that actually decides whether it should ship.
Four gates, not a vibe check
Each gate below takes decisions the pilot has already made, not a new dataset, and asks one narrow question of them. None of these tells you the pilot is safe. Each one narrows what "almost ready" actually means down to something specific enough to act on.
The margin gate: how close are the calls, not just whether they're right
A pilot dashboard reports a pass rate. It never reports how close the calls were. Two decisions can both land on the correct answer and be completely different animals: one wins by a mile, the other barely edges out an alternative that would have been wrong.
Start with decisions the pilot already made. Where comparative scores or the alternatives considered were preserved, inspect the margin directly. Most stacks don't keep that, so more often you'll need to manufacture the signal: rerun a case with small perturbations, or repeat it a few times, and see how often the decision flips.
# Start with decisions the pilot already made.
# Use whatever evidence is available: preserved comparisons if you have
# them, controlled perturbations or repeated runs if you don't.
for d in pilot_decisions:
if the_decision_appears_fragile(d):
flag(d, "thin margin")You're not grading whether the pilot got it right. You're finding out which of its correct answers were confident and which ones were lucky, because production is what turns "barely correct" into "wrong," on a day nobody's watching closely.
The explanation gate: how long does it actually take to explain a miss
Pick one wrong or borderline output from the pilot. Now, honestly, how long does it take someone on the team to explain why the system did what it did, using only what's already logged.
How long does the honest answer actually take. Photo: Castorly Stock on Pexels
If the answer arrives in a few minutes, that's real evidence. If it takes an afternoon of digging through prompts and traces, and the person doing the digging is the one engineer who built the thing, that's not a bug to fix before shipping. That's the actual blocker the "almost ready" meetings keep circling without naming.
The drift gate: does it hold up past the clean conditions of the demo
Pilots often run under cleaner conditions than production: curated inputs, shorter interactions, closer supervision, or a happy path chosen for demonstration. Take a handful of the pilot's real decisions and push them through conditions the demo never included: a multi-turn conversation, an ambiguous ask, a case where something confirmed three turns ago quietly stopped being true.
# Same decision logic, conditions the demo never tested.
for scenario in drift_scenarios: # built from real conversation shapes, not the demo script
decision = pilot.decide(scenario)
if decision.assumed_stale_context_was_still_valid:
flag(scenario, "didn't hold past turn one")A pilot that only gets exercised on the conditions someone chose to demo has never actually been tested. This is where you find out.
The catch gate: will anything notice before a customer does
The first three gates ask about the decision itself. This one asks what happens after. If this decision ships wrong, is there anything positioned to notice a monitor, a review step, a person before someone outside the company does?
If the honest answer is "we'd find out from a complaint," this gate fails regardless of how good the model is underneath it. A system can be highly capable and still have zero ability to catch its own mistakes, and those are two separate problems that get solved two separate ways.
The four gates at a glance
| Gate | Asks | Fails when... |
|-----------------|------------------------------------------------------------|----------------------------------------------------------|
| **Margin** | How close was the decision, not just whether it was right? | Correct answers are winning by a hair, unnoticed |
| **Explanation** | How long to explain a miss, using what's logged | It takes hours, and only one person can do it |
| **Drift** | Does it hold up past the demo's clean conditions | It only works on inputs someone chose to show leadership |
| **Catch** | Will anything notice before a customer does | The honest answer is "a complaint" |What this checklist doesn't give you
Be honest about the edges, or a clean run through these four gates reads as more certain than it is.
A clean pass is evidence, not a guarantee. All four gates can come back green on the sample you checked, and something outside that sample can still go wrong. This narrows the search. It doesn't close it.
The margin gate doesn't apply cleanly to genuinely judgment-heavy work. If there's no single correct answer to measure a runner-up against, forcing a margin score onto the decision manufactures false precision. For that kind of work, lean harder on the explanation gate: can the reasoning be reconstructed and defended, not whether there was a numerically closer alternative.
This is a snapshot, not a standing guarantee. A pilot that clears all four gates the week of the review still needs to clear them again once it's actually running unsupervised. Behavior drifts. The review doesn't.
None of this replaces the person accountable for the decision. These gates tell you where to look. They don't tell you whether the risk is worth taking. That is still a judgment call, and it should stay one.
Where to start this week
Pick the pilot that's been sitting in "almost ready" the longest. Find its highest-stakes decision point, the one closest to an actual consequence: a message sent, a ticket closed, a refund approved. Run just the explanation gate against it: grab one wrong or borderline output, and time, honestly, how long it takes someone to explain why it happened, using only what's already logged.
That single number is usually the real reason the pilot hasn't shipped. Add the other three gates once that first one surfaces something worth chasing.
Doing this by hand, once, on one pilot, is the whole exercise this week. Running these four gates continuously, on every decision, before every go/no-go meeting instead of during one review that happens to catch a good week, is the harder problem, and it's the direction I'm building Nalyqor toward.
But you don't need it to run the version above. Pick the pilot, pick the decision, and time the honest answer.