A payables agent at a mid-sized company reads each incoming invoice, matches it to a purchase order, and either releases the payment or sends it to a person. This is a composite, built to show the shape rather than report a case.
On a Tuesday, someone in finance edits the guidance document the agent reads and adds a line to speed up onboarding for new vendors. The edit is reasonable. It also changes what the agent treats as a known vendor.
Nothing breaks. Error rate is flat, latency is flat, every payment call returns a success. Over the next three weeks the agent releases payments to vendors that used to go to a person. Someone spots it at month-end reconciliation, by which point some of the money has moved and some of it can't easily be pulled back.
Every alert the team owned was working. They all fire on things going wrong, and nothing went wrong in any way those alerts could see. The decision became risky on Tuesday. The first signal arrived about three weeks later.
An alert can only be as early as what it watches
An error-rate alert watches execution, so it fires when execution fails. A complaint-volume alert watches customers, so it fires after customers are affected. Both are fine instruments. Neither can be early about a decision that executed cleanly.
"Before they ship" needs a definition. A decision ships when it becomes an action outside the system: the payment released, the email sent, the record changed. Before that point you often can't know whether the decision was right in the world, because the outcome hasn't happened yet.
You can know whether it's the kind of decision that deserves a look: consequential, hard to reverse, and carrying something new. So proactive alerting trades a question you can't answer yet (was it wrong?) for one you can (was it risky?). That trade is the whole technique. It's why these alerts can fire early, and why they're noisier than an error alert.
Three signals that exist before the consequence
Two of them are alerts and one is a gate.
A change alert. It fires when something that shapes a decision changes: the model version, a provider default, a prompt or tool description, a document the agent retrieves. It doesn't need the decision to look wrong. In the payables case it would have fired on Tuesday, naming the guidance document and the approval decision that reads it.
This works because many behavior shifts start with a change event you can see at time zero, while the effect may not surface for weeks. Not all of them do. The mix of inputs and the data outside your system move without anyone editing anything, which is what the next alert is for. If you wrote the list of what can change a decision without a deploy (item 2 in the readiness checklist), you already have the inputs. The alert itself can be a message in a channel that says what changed and who owns the decisions it touches.
Most changes are harmless, so the message means "go look at the decision this touched," and it says little about whether anything is wrong.
A shift alert. It compares how a decision splits now against the baseline you wrote down before launch: the share of invoices paid without review, how often refunds are approved, how often the agent escalates. It fires on a change in the population, so it doesn't need any single decision to have been confirmed bad. In the composite, the share paid without a person would have moved within days of the edit.
It's noisier than the change alert, and it has two ways to mislead you. The baseline has to come from a comparable period, because month-end really does look different from mid-month. And a shift is a reason to look, not a verdict. The agent may be responding sensibly to a different mix of invoices.
A hold. This is the only one of the three that acts before the action. For a narrow class of decisions, the ones that are consequential and hard to reverse and also meet a risk condition, the action waits for a person. The condition might be a vendor the system has never seen, an input unlike anything in the pilot, or a call that sits close to the line. A hold isn't really an alert. It's a gate with a queue behind it, which is why it should be small.
Signal | Fires when | How early | What it costs you |
|---|---|---|---|
Change | Something that shapes the decision is modified | Before the first affected decision | Attention cost: most changes will be benign |
Shift | The decision's split moves against its baseline | After a few decisions, before the complaints | A fair baseline; busy periods look like drift |
Hold | A consequential, hard-to-reverse call meets a risk condition | Before this one action | A queue a person has to keep clearing |
What this doesn't give you
It doesn't catch a decision that was wrong from the first day. If the agent launched with a bad reading of the policy and never changed, no change event fires, the split never moves, and the baseline you compare against has the mistake built in. These signals find movement and risk. They don't find error.
A hold queue can quietly turn into a rubber stamp. If it fills faster than someone can think about each item, people start approving to clear it, and the gate stops being a check while still looking like one. When the queue grows, tighten the condition. Don't add reviewers and hope.
Alerts that nobody owns are dashboards. Before you build any of this, check whether the decision needs it at all. For something cheap to reverse, "we'd hear about it from a customer" is a legitimate answer, as long as you chose it deliberately. Reversibility decides how much of this is worth the effort.
One thing to do this week
Pick the decision whose bad version would cost you most. Write one sentence about it in this form:
When ___ changes, ___ looks at ___ before ___ happens.
If you can't fill in a named person, or you can't say what happens next that makes it too late, you have a dashboard and not an alert. Then build the change alert for that one decision. Post a message to a channel when the model version, the prompt, or a document it reads is modified, addressed to whoever answers for the decision. It needs no baseline and no judgment calls, so it's the cheapest of the three.
Start there unless this decision already deserves a hold. If the action is consequential and hard to reverse enough that it ought to need approval, don't wait on the gate. Add the shift alert once you have enough decisions for a useful baseline, and after a month, check how many of the change alerts turned out to matter.
Where this goes
Done by hand, this is one decision and one sentence. Keeping change, shift and hold current across every consequential decision a system makes, while the things that shape those decisions keep moving, is the part every team ends up rebuilding for itself. That continuous version is the direction I'm building Nalyqor toward. You don't need it to write the sentence.
