Putting a Number on a Bad AI Decision: A Cost-of-Error Worksheet

I wrote recently about where the money from a bad AI decision actually goes: six places, landing on six different clocks, under owners who have no reason to connect any of it back to the moment a system chose wrong. That piece named the six. It didn't tell you how to go find them on a decision sitting in your own logs right now.

This is that worksheet. Six rows, one incident, an honest answer about which rows you can actually fill in and which one you can't.

Use the same case that piece did. The worksheet doesn't care whose incident it is. It only cares that you can trace it back to one decision. You probably already have your own version of it: a support agent reads an ambiguous billing message as a dispute instead of a credit request, routes it to the heavier workflow, and everything downstream executes correctly on the wrong premise.

Pull up a decision like that: a specific one, timestamped, that you already know went sideways. Not a category. One instance.

The six rows aren't equally hard to fill in, and treating them that way is how this exercise stalls before it starts. They sort into three tiers.

Already on your bill. The decision itself, and its recovery loops. You don't have to go looking. You have to know to look.

Reconstructable from data you already have. Downstream execution, detection, correction. None of it is sitting in one place, but all of it already exists somewhere, filed under the wrong name.

Not a number. Forfeited scope. The worksheet asks for it anyway, and the honest answer is often a sentence, not a figure.

Work the tiers in that order. The first tier takes five minutes. The second is where the real digging happens. The third is where most people quietly stop, which is fine, as long as the row stays visibly blank instead of quietly dropped.

Row 1, the decision. What to write down: the cost of the call or calls that produced the wrong choice. Where to find it: your provider bill, filtered to this trace or session ID. It's already there, denominated in the one currency the invoice understands, and it's the only row here that never requires estimation.

Worked example: the misroute call itself. A few cents. The number every cost dashboard already reports, and the least interesting number on this page.

Row 2, the recovery. What to write down: retries, re-retrieval, anything the system did a second time trying to settle on this decision before landing on the wrong one. Where to find it: the same bill, the same trace, filtered to calls after the first. If nothing retried, this row is zero, and zero is a real answer here, not a placeholder.

Worked example: none. The misroute happened cleanly on the first pass. Zero, and worth noting anyway, because a decision that costs nothing to make twice is exactly the kind that's easiest to miss entirely.

Row 3, downstream execution. What to write down: everything that ran correctly on the wrong premise. Where to find it: this is the hard part, and it's the same hard part from the cost audit I wrote up a few weeks back: joining spend to a unit of work using whatever ID already threads a request to a trace. If you've already run that join, you have the raw ingredients. What's missing here specifically is the second join, from the downstream trace back to the decision that set it running, and most stacks don't preserve that link at all. Where it isn't preserved, don't chase a single dollar figure. Price a class instead: pull every case that shares the same ambiguity, not just the one in front of you.

Worked example: the disputes workflow running end to end on a request that was never a dispute. Account history pulled, a verification pass run, a response drafted with the wrong tone and the wrong obligations. Multiply by every ticket that shared the same ambiguity in the same window, because the interesting cost was never the one ticket.

Row 4, detection. What to write down: the engineering hours spent working backward from a symptom to this decision. Where to find it: your incident tracker already has this, filed under whatever symptom got reported, not under the decision that caused it. Recovering the row means re-attributing hours you already logged, not collecting new data.

Worked example: the customer wrote back confused, and someone spent forty minutes reading transcripts before finding the misroute. Forty minutes, at whatever that engineer's time is worth, already sitting in a ticket labeled "confused customer," not "misrouted decision."

Row 5, correction. What to write down: the one-time fix, plus whatever guardrail got added afterward and now runs forever. Where to find it: the fix is on the same incident record. The standing cost is different in kind, not just harder to find: it's the latency and complexity the new guardrail adds to every future call of that type, and it won't show up as a line item anywhere, only as a slightly slower version of a workflow that used to be simpler.

Worked example: a disambiguation prompt added before the routing decision. One afternoon to write and ship. Then a small, permanent tax on every ticket that passes through routing from now on, whether or not it was ever going to be ambiguous.

Row 6, forfeited scope. What to write down: what the team quietly stopped delegating to the system after this. Where to find it: nowhere, and that's the point. This is the one row you cannot fill in with a number, on purpose, and leaving it blank isn't a mistake in the worksheet. It's the worksheet telling you the truth. Write a sentence instead: what does the team now route to a human that it used to route to the system? If the honest answer is nothing, that's real information too.

Worked example: after two of these, the team pulled ambiguous billing language out of the auto-router entirely and sent it to a human queue instead. That queue isn't free. Nobody bills it against the misroute that caused it, and probably never will.

Add rows 1 through 5 and you get a number. Don't present it as the cost of the decision. Call it what it is: a floor, built from whatever fraction of the six you could actually see, and that fraction changes decision to decision depending on what your stack happens to preserve. The useful output here was never a precise total. It's a ranking: which decision patterns are worth a closer look, priced roughly enough to sort by, not exactly enough to defend in a budget meeting.

Row 6 never joins the sum. It has no unit that would let it. Its job is to sit there, visibly blank or visibly answered, so nobody mistakes the number above it for the whole cost.

Pick one decision, not a category: one timestamped instance you already know went wrong. Fill in rows 1 and 2 first. They take five minutes, and they're already on your bill. Then do row 4, because your incident tracker already has it too, just filed under the wrong name. Stop there for a first pass. Rows 3 and 5 take real digging, and row 6 takes an honest sentence, not a lookup. Three rows and twenty minutes is enough to know whether this decision pattern is worth the other three.

Doing this once, by hand, on one decision, is a worksheet. Doing it on every decision a system makes, continuously, without someone filling in six rows by hand each time, is closer to what I'm building with Nalyqor: a way to keep rows one through five joined to the decision that caused them automatically, so the ranking updates on its own instead of needing someone to rebuild it every time a bad call is worth investigating.

But the worksheet works without it. Pick the decision, open six rows, and see how much of your own accounting you can actually close.