Catching Wasted Spend: A Practical AI Cost Audit

You ship the obvious fix. The code-review agent runs on every pull request, it's the biggest line on the AI bill, and it's been running on a frontier model since the week someone prototyped it. So you move the high-volume path to a mid-tier model. Same prompt, same workflow, roughly sixty percent off the per-call price. The bill drops the following week. Somebody puts it in the Friday update.

Two months later the bill is back where it started, and nobody can point at the thing that undid it.

I've written before about where AI money pools and what a single bad decision actually costs. Both of those find spend. Neither one tells you what happens after you act on what you found, which is the part almost nobody checks, and the part where most of the savings quietly go back.

A cost fix operates on price. It makes a unit of work cheaper. What it does not touch is the number of units the system decides to do, or the quality of the decision behind each one, and those are the two things that actually determine your spend.

So when a fix degrades decision quality even slightly, the work has to go somewhere. The system retries. A person re-runs it. Someone quietly takes the task back. Or the cheaper unit cost gets absorbed by growing volume. The work reappears, priced differently, filed under a different name, sometimes on a completely different budget. The invoice shows a win because the invoice only ever sees the part of the work that ran on a model.

A cost audit asks where the money went. A recovery audit asks whether it left.

That's the whole reframe, and it changes what you measure. Not the bill. The same unit of work, before and after, judged on what it took to produce a good outcome.

Pick one cost change you've made in the last quarter: a model swap, a trimmed context window, a caching layer, a removed reasoning step. You need one task type it touched and a window of traffic on each side of it. If you already did the join from the first-pass audit, tying spend to a unit of work rather than to a token, you have everything you need. If you didn't, do that first, because none of this reads correctly per token.

None of these four produces a verdict on its own. Each one tells you whether a particular escape route was taken, and the routes are not exclusive. Money can leave by more than one door at once.

Compare how much effort a completed task of this type takes now versus before. Model calls per finished task, retries, re-retrievals, repeated passes over the same input.

This is the loop tax coming back through the side entrance. A cheaper model that's slightly less decisive gets asked more times. Sometimes the system asks: a validation step that fails and retries, a planner that takes another lap. Sometimes a person asks, by re-running the agent because the first output looked thin, which never appears in any config change and shows up only as a call count.

In the code-review case, this is where the first chunk of the sixty percent goes. The unit price fell and the unit count rose, and unless you were watching calls per pull request rather than cost per call, nothing about that transfer was visible.

Compare what the task type produces now versus before. Not the pass rate on your evals: the disposition of the real output. How much of it gets accepted, dismissed, escalated, reversed, or corrected downstream.

This one is slower to show up and more expensive than the first, because it isn't a cost signal at all until you go looking for it. A review agent that flags more things nobody acts on hasn't got more expensive per call. It has got more expensive per useful comment, and it's spending human attention rather than tokens, which is the silent-failure tax with the polarity flipped: not a wrong answer that looked right, but a right-looking output that bought less than it used to.

Where the acceptance signal already exists somewhere in your stack, a resolved comment, a merged suggestion, a closed ticket, use it directly. Where it doesn't, sample by hand. Forty reviewed outputs from each side of the change will tell you more than a month of aggregate spend.

Ask what people stopped delegating after the change. This is the check with no query behind it, and it's usually the largest number on the page.

When an automated step gets less reliable, the adjustment is rarely a ticket. It's a senior engineer who quietly starts reading the diffs on the files that matter, because the agent's read stopped being worth trusting on those. Nobody logs that. There's no config commit for it. The workflow still exists on the architecture diagram and the bill still shows it running, cheaper than before, on the subset of work where it doesn't matter much.

You find it by asking. One question in a standup, to the three people closest to the workflow: what do you now do yourself that you used to let this handle? If the answer is nothing, that's a real result and a good one. If two of them name the same thing, you've found the money.

Compare cost per completed outcome, not total spend, across the same window. This is the only check that can clear a fix rather than convict it.

A flat bill after a real fifty percent saving means volume doubled and the saving is genuinely there, invisible, holding down a number that would otherwise have gone up. A flat bill after a fix that changed nothing per outcome means the fix didn't work and growth is covering for that too. On the invoice, those two situations look identical. On cost per outcome, they look nothing alike, and this is the reason step zero of the first audit was to change the denominator: the total was never the measurement. It was the thing hiding the measurement.

Run the four and you get something more useful than a savings figure: a straight answer on whether the fix reduced what the system spends to produce a good outcome, or only what it spends. Those are different results and only one of them survives the next quarter.

You probably don't have a clean before. Almost nobody captures a baseline ahead of a cost change, because at the time it was a config edit, not an experiment. Don't reconstruct one out of aggregate history and pretend it's comparable. Run the comparison forward instead: hold a slice of the workflow on the old configuration for a window and read both sides live. It costs a little money to learn whether you saved any.

Attribution stays soft. Two months is long enough for the model version, the traffic mix, and three unrelated prompt edits to move underneath you. Treat a finding as a workflow worth investigating, not a proven cause, and be more suspicious of the checks that confirm what you hoped.

Cheaper is sometimes just correct. Plenty of fixes work exactly as intended. The point of the fourth check is to be able to say so with something behind it, instead of pointing at a bill that would have looked the same either way.

Take the last cost change you shipped, and run checks one and three on it. Nothing else.

Check one is a count you can pull this afternoon: model calls per completed task of that type, this month against the month before the change. Check three is a question you can ask three people in ten minutes. Between them, they cover the two places savings most reliably go, one on the bill and one off it entirely, and if both come back clean, the fix is very likely real.

Doing this once, on one change you already made, is a recovery audit. Doing it on every change, continuously, so that a saving is provisional until the system proves the work didn't just move, is the harder version, and it's the one every team currently rebuilds by hand or skips. That's the direction I'm building Nalyqor toward, on the belief that whether AI spend bought a better decision is a question that should be answered automatically rather than reconstructed by whoever remembers to ask.

You don't need it to run the four checks. You need one change, one task type, and the willingness to find out that the part that bought nothing is still there, wearing a cheaper name.