Evaluating AI agents: how to know an agent actually works

There is a version of the demo trap that is specific to agents, and it is worse than the chatbot version. A chatbot gives a slick answer once and you are impressed. An agent completes a whole task once, books the thing, fixes the bug, processes the refund, and you are convinced. Then you ship it, and it turns out the run you watched was one of the good ones. The agent that dazzled in the demo takes a different, broken path on the next input, and because it acts rather than just talks, the failure is not a bad sentence; it is a bad action on your systems.

Evaluating an agent is a genuinely different problem from evaluating a model, and most teams reach for the wrong tools. This post is why agents break the usual eval playbook, the dimensions that actually predict whether an agent works, and how to build an evaluation you can trust. It is the “how do I know it works?” companion to agents versus workflows and what governed agentic AI means.

Why agents break the usual eval playbook

Everything that makes an agent useful also makes it hard to evaluate. Unlike a single-turn model, an AI agent:

• takes many steps, so there is a whole path to judge, not one output;
• is stateful, carrying context and side effects from step to step;
• uses tools and acts on the world, so a mistake changes real data, not just the reply; and
• is non-deterministic, so the same task can go well or badly on different runs.

That last point is the killer. With a chatbot, a right answer is decent evidence the system works. With an agent, a right answer might be luck that will not repeat: two agents can reach the same correct result, one by a clean and correct process and one by a lucky accident that falls apart on the next, slightly different input. If you only grade the final result, you cannot tell those two apart, and you will ship the second one.

Grade the journey, not just the destination

The central shift in agent evaluation is from scoring the answer to scoring the trajectory, the actual sequence of steps and tool calls the agent made to get there. In practice, useful evaluation happens at three layers:

• Final outcome: did the task actually get done, correctly, end to end?
• Trajectory: was the path sound, the right tools, in a sensible order, with the right arguments, correctly interpreting the results, and recovering from failed calls, or did it stumble to the answer with wasted or dangerous steps?
• Per-step, in production: is each real turn behaving as expected once live?

Trajectory evaluation is what catches the process problems that a final-answer check hides: the agent that got the right number by calling the wrong tool and getting lucky, or the one that succeeded while making three unnecessary writes to your database along the way. The destination can look fine while the journey is quietly broken.

The dimensions that matter

There is no single “agent score.” As with the wider metrics that actually matter, you measure a set, and you slice it by the tasks you genuinely care about:

• Task success. Did it complete the goal, end to end, not just produce plausible-looking steps? This is the headline, but it is not enough on its own.
• Trajectory quality. Did it plan sensibly, pick the right tools in the right order, pass correct arguments, read results correctly, and recover from errors?
• Tool-use correctness. Valid calls, correct parameters, correct interpretation of what came back. Most agent failures are tool failures.
• Efficiency and cost. How many steps, how much latency, and how much token spend did it burn to get there? A correct agent that takes forty steps and a fortune is not a viable agent.
• Reliability. Does it succeed consistently across repeated runs, or only sometimes? More on this below, because it is the one people underrate.
• Safety and containment. Did it stay inside its permissions, avoid actions it should not take, and stop when it should? For an agent that can act, this is not optional.

Reliability is the dimension everyone underrates

Because agents are non-deterministic, a single successful run is weak evidence. The question is not “can it do this task?” but “how often does it do this task correctly when I run it again and again?” This is why serious agent benchmarks have moved to reliability-style metrics: rather than reporting whether an agent passed once, they report whether it passes repeatedly across independent trials. An agent that succeeds nine times in ten is a very different proposition from one that succeeds once in three, even though both can produce a flawless demo.

Measure reliability by running the same tasks many times and looking at the distribution, not the best case. An agent you are about to hand real work to should be judged on its bad runs, not its good ones.

The methods, and where each falls short

• Offline benchmark suites give you repeatable, comparable scores on fixed tasks. Good for regression-catching and model comparison, but fixed tasks go stale and can be gamed (see below).
• LLM-as-a-judge uses a model to grade outputs that have no single correct string. Scalable and genuinely useful, but it has real weaknesses: position bias, sensitivity to prompt wording, a tendency to over-score, shaky fine-grained scoring, and an inability to check claims against what actually happened. Research finds judges are weakest exactly where agents fail, on verifying evidence and multi-step reasoning. Use one only after validating it against human labels, keep its verdicts coarse, and distrust a 100% pass rate. (This deserves its own treatment, and will get one.)
• Human evaluation remains the ground truth for judgement-heavy tasks, and the thing you validate your automated judges against. Accurate, expensive, not scalable on its own.
• Simulated and sandbox environments let an agent act without touching real systems, which is essential for safe, repeatable testing. The trade-off is ecological validity: an agent that shines in a stable simulator can transfer poorly to the messier real world.
• Online and production monitoring is the only method that measures the agent on real traffic. It is where you find the failures no offline suite anticipated, and it is not optional.

The theme: every method has a blind spot, so a credible agent evaluation combines several rather than betting on one.

What the public benchmarks tell you, and what they don’t

Benchmarks are useful reference points, and it helps to know what each actually measures:

• SWE-bench (and the cleaner, human-verified SWE-bench Verified) grades software-engineering agents on real GitHub issues, by whether the repository’s test suite passes after the agent’s patch.
• GAIA measures general-assistant tasks that require multi-step reasoning and tool use, like a capable person with a browser and a code interpreter.
• WebArena puts an agent in a live browser and asks it to complete realistic tasks across sites like shops and forums.
• tau-bench tests tool-and-policy interactions: the agent talks to a simulated user, follows domain rules, and is scored against the resulting system state, with an emphasis on reliability across repeated trials.

Treat all of them as proxies, not verdicts. Public benchmarks have two well-documented problems. Contamination: popular test sets leak into training data, so a high score can reflect memorisation rather than capability. Reward hacking: agents find ways to pass the check without doing the task, exploiting the test harness rather than solving the problem. And there is a persistent gap between benchmark performance and real deployment: an agent’s leaderboard score is not its production reliability. The only benchmark that fully counts is your own agent, on your own tasks, in something close to your own environment.

How to build your own agent evaluation

You do not need a research lab; you need discipline. A workable agent evaluation looks like this:

1. Build an eval set from real tasks, including the awkward and adversarial ones, not just the happy path you demoed.
2. Score the trajectory, not just the outcome. Capture the steps and tool calls so you can grade the path, and define what “a good path” means for each task.
3. Measure reliability by repetition. Run each task many times and judge the agent on its distribution of outcomes, not its best run.
4. Instrument everything. Structured logs of every step, tool call, argument and result, so you can reconstruct and score any run after the fact. This is also the backbone of your evaluation harness.
5. Validate any automated judge against human labels before you trust it, and keep its scoring coarse.
6. Evaluate offline and monitor online. Catch regressions before launch, then keep watching, because agents drift as inputs, tools and models change.

Evaluation is how you govern an agent

This is where evaluation stops being an engineering nicety and becomes governance. An agent that can act on your systems is only safe to deploy if you can show it behaves, reliably and within bounds, and that evidence is exactly what meaningful human oversight needs to function: an overseer can only intervene sensibly if the agent surfaces what it is doing and how confident it is. The trajectory logs, reliability figures and safety checks you build for evaluation are the same records that make an agent assurable, the evidence that lets a customer, a board or a regulator trust an autonomous system with real work. Evaluating an agent well and governing it well are, in the end, the same activity.

The short version

Evaluating an AI agent is not like testing a chatbot, and using chatbot methods is how teams ship agents that dazzle once and break in production. Because agents take many steps, act on the world and behave non-deterministically, a right final answer is weak evidence. Grade the trajectory, not just the destination; measure the dimensions that matter (task success, trajectory quality, tool-use correctness, efficiency, reliability and safety); and take reliability across repeated runs seriously, because a single good run is often luck. Combine methods, offline benchmarks, validated LLM judges, sandboxes, human review and production monitoring, since each has a blind spot, and treat public benchmarks as proxies rather than proof. Do it well and you get two things at once: an agent you can trust, and the evidence that lets everyone else trust it too.

Building or deploying agents and need to know they will hold up? Our AI agent development builds evaluation and guardrails in from the start, and our AI governance consulting turns agent evaluation into assurance you can put in front of a customer or regulator. For the wider picture, see what governed agentic AI means.