Evaluating AI Behavior Tools: A Buyer's Checklist

You take three calls in a week. Three tools, three dashboards, three fluent walkthroughs of an agent failure, each one ending with a timeline that turns red at exactly the right moment. All three demos are good. All three are running on data the vendor chose. By the third one you can't articulate what separates them, and the feature pages don't help, because in a category this young every feature page says the same four things.

I wrote on Tuesday about why this category is forming at all: enough teams are hand-building the same internal tool that the patches have started looking like drafts of one product. That piece argued the buy-versus-build question is now real. It didn't say how you'd actually judge what's on the table.

Fair warning before the checklist: I'm building one of these. Read the questions with that in mind. They're the ones I think a buyer should ask, which means they're also the ones I have to answer.

Every tool in this space can render what happened. That part is genuinely solved, and it's what a demo is built to show off, because a timeline with a red block in it is the easiest thing in the world to make legible in twenty minutes.

What you're actually shopping for is the layer above it: whether the tool can tell you why the system chose what it chose, and whether that choice was sound. You cannot evaluate that on a failure the vendor picked, because the vendor picked it precisely because their tool explains it well.

So the whole thing rests on one artifact, and it isn't theirs.

Pick a failure you already diagnosed. Something from the last quarter, closed, understood, where you know exactly what went wrong and how long it took someone to figure it out. Write down the answer somewhere the vendor can't see it.

You're not asking the vendor to solve the incident. You're asking whether the explanation they produce matches the one you already know is correct. That's the entire mechanism, and it's why this works when a feature comparison doesn't: you're grading against a key.

Then run the demo on that.

A demo shows you what a tool can display. Your own incident shows you what it can explain.

Bring your own failure. Everything below is just what to ask once you have.

Each of these is a test, not a feature request. Feature lists converge in an unformed category. What people can do with your data does not.

The first question is procedural and it filters more aggressively than anything else on this list. Can they take your traces, or a sanitized slice of them, and show you their tool working on your failure inside the evaluation window?

What a real answer sounds like: a scoped proof of concept on your data, with a named list of what they need from you to make it work and an honest estimate of how much of your incident they expect to be able to reconstruct.

What a deflection sounds like: onboarding happens after signature, the sample dataset is representative, or a long explanation of why your stack is unusual. Sometimes that last one is true. It's also exactly what you'd say if the tool only worked on data shaped like the demo.

Point at a single decision in your incident, the one you know was the wrong turn, and ask what the system was weighing at that moment. Not what it did next. What else was on the table, and what made it pick this.

What a real answer sounds like: a clear explanation of what information the tool can observe at decision time versus what it reconstructs afterward, and a straight answer about which instrumentation you'd need to add to move something from the second column to the first. Honest limits here are a good sign, not a bad one. Most of this information is never written down at the moment it exists, and a vendor who says so understands the problem.

What a deflection sounds like: a fluent paragraph that narrates your trace back to you. This is the most common failure mode in current tooling, and it's convincing right up until you check it against what you already know went wrong. A summary of a trace is still a trace. It reads like an explanation because it's written in sentences, but nothing in it came from anywhere except the steps you could already see.

You know how your incident actually surfaced. A customer wrote in, or a number moved, or someone noticed something odd on a Tuesday. Ask where in that timeline the tool would have said something.

What a real answer sounds like: a specific signal, a specific point on your timeline, and a willingness to say "we would not have caught this one" about at least some class of failure. Ask what that class is. That answer tells you more about the product than the demo did.

What a deflection sounds like: anomaly detection on volume, latency, or error rate. Those are real signals and you probably already have them. A behavior that stayed inside every operational threshold and was still wrong is the entire reason this category exists. If the alerting story is thresholds on metrics you already collect, you are being sold the previous category in new language.

Not the license. The standing cost of operating it: what fraction of decisions get evaluated, what the evaluation itself spends in tokens and latency if there's a model in the loop, what instrumentation you have to add and maintain, and how many alerts a week land on a human who has to triage them.

What a real answer sounds like: a sampling strategy, a rough number, and a clear statement of which parts run on your infrastructure versus theirs. Anyone who has run this at scale has been forced to think about it.

What a deflection sounds like: negligible. A tool that judges the quality of AI decisions is itself doing work per decision, and that work has a bill and a false-positive rate. "Negligible" usually means "not yet measured at your volume."

Take the output for your incident and hand it to a person who didn't build the system: an engineer from another team, a support lead, whoever ends up in the room when this goes wrong for real. Ask them what they think happened.

What a real answer sounds like: they get it roughly right without you narrating. The artifact stands on its own.

What a deflection sounds like: an interface that only makes sense to someone who already knows what they're looking for. That's the same failure as the trace itself, one layer up and better designed. A record you have to already understand isn't an explanation; it's a nicer place to keep the question.

Not a recap. This is the version to have open during the call, so a deflection is recognizable while it's happening rather than three days later.

Then run the same five on the thing you already built. That's the part most teams skip, and it's the only way the buy-versus-build question gets an honest answer. The comparison is never a vendor versus nothing. It's a vendor versus the internal tool you've already built, whether you call it a tool or not. Almost nobody has scored that one, because it was never presented to anyone. It just accumulated.

Score it anyway. Some teams will find their homegrown version answers question 2 better than anything on the market, because it was built by people who knew exactly which decisions mattered. Those teams should keep it. Most will find it answers question 1 and nothing else.

One incident is a probe, not a benchmark. A tool that whiffs on your one case may be strong across the ninety-nine you didn't bring. Two or three incidents with genuinely different failure shapes, one bad judgment call, one slow degradation, one thing that only broke under load, is a much better sample and still not a large one.

It's weighted toward explanation over prevention. Four of the five questions are about understanding a failure after it happened. A tool that's mediocre at explaining and genuinely good at catching problems before they ship will score badly here and might still be the right purchase. Know that you're measuring one axis, and add your own question if prevention is what you're actually buying.

You are partly buying a roadmap, and the roadmap is where the risk is. In a category this young, every honest vendor's answer to something on this list is "not yet." Scoring purely on what exists today selects for whoever formalized their shape earliest, which is not the same as whoever formalized the right one. So ask one question that isn't on the checklist: which of these five do they think matters most, and why? You learn more from whether they've thought about it than from the answer.

Before you take another call, build the packet. Pick one closed incident. Write three lines: what the system did, what it should have done, and how you eventually figured that out. Sanitize whatever needs sanitizing. Put the answer in a file the vendor doesn't get.

That's thirty minutes, and it changes what a demo is. You stop watching a product and start testing one, and you can do all of it before you have any idea which tool you want.

The harder version of this problem is that the five questions don't only apply to a purchase. They're the same five questions any team hand-building this layer has to answer for itself, permanently, on every decision the system makes rather than on one incident during one evaluation window. That's the gap I'm building Nalyqor to close, on the belief that a category currently being reinvented separately inside dozens of engineering orgs is one somebody should build once.

But the checklist doesn't need a product behind it. Pick the incident, bring it to the next call, and see how much of it the tool can actually explain.