Eval metrics that actually matter (beyond accuracy)

Ask a team how their AI system is doing and you will usually get one number back: accuracy. It is the metric everyone reaches for, and it is the one that hides the most. A system can be “90% accurate” and completely unsafe to ship, because that single figure tells you nothing about which 10% it gets wrong, how confidently it was wrong, or whether “correct” even means anything for the task it is doing.

If you have already built the evaluation harness, the next question is what to actually measure with it. This post is the set of metrics that predict how an AI system behaves in the real world, and why accuracy alone will let a dangerous system through.

Why “accuracy” is the wrong headline number

Accuracy is not useless. It is just doing too much work as a single number, and it fails in three specific ways.

• It collapses different failures into one figure. A model that is wrong on low-stakes edge cases and a model that is wrong on your highest-stakes cases can post the same accuracy. The number treats them as equal. Your users will not.
• It ignores confidence. Being wrong and knowing you might be wrong is recoverable. Being wrong while sounding certain is how bad outputs get actioned. Accuracy says nothing about the difference.
• It assumes a single correct answer. For summarisation, drafting, extraction with judgement, or anything open-ended, there is no one right string to match against. Exact-match accuracy on those tasks measures the wrong thing precisely.

So accuracy is where you start, not where you stop. Here is what to measure instead, or alongside it.

Faithfulness: is the output actually supported?

For any system that answers from source material, retrieval-augmented generation, document Q&A, summarisation, faithfulness (also called groundedness) is the metric that matters most. It asks a simple question: is every claim in the output actually supported by the source it was given, or did the model invent it?

This is the metric that catches hallucinations, because a hallucination is just a fluent answer that isn’t grounded. A response can be perfectly relevant, well-written, and confidently wrong. Accuracy against a reference answer might even score it highly. Faithfulness against the retrieved context is what exposes it.

You measure it by decomposing the output into individual claims and checking each one against the source, usually with a model-based judge. Which raises the obvious question: can you trust the judge? Only if you have validated it against human labels first, which is a whole discipline of its own and one we will cover separately. For now the point stands: if your system reads from sources, faithfulness is not optional.

Relevance and completeness: did it answer the question?

Faithfulness stops the model making things up. It does not check that the model answered the actual question. Two more metrics cover that:

• Relevance. Is the output on-point for what was asked, or is it accurate but beside the point? A grounded, faithful answer to a question the user did not ask is still a failure.
• Completeness. Did it cover what it needed to, or leave out something material? In production RAG the common failure is not a wrong answer but a partial one, right as far as it goes, missing the clause that changes the decision.

Together, relevance and completeness are the difference between “technically correct” and “actually useful,” and they need measuring separately because a system can be strong on one and weak on the other.

Safety metrics: refusal, toxicity, leakage

Everything above measures whether the system is right. Safety metrics measure whether it is harmful, which is a different axis entirely. Depending on what the system does, you want to track:

• Refusal quality. Does it decline the things it should decline, and, just as important, does it avoid over-refusing safe requests? Both directions are failures.
• Toxicity and bias. Does the output stay within bounds across the range of users and inputs it will actually see, including adversarial ones?
• Data leakage. Does it keep system prompts, other users’ data, and information it was told to withhold from ending up in the output?

These are the metrics that show up in a risk classification and that regulators care about, because they map directly to harm rather than to quality. A system can be highly accurate and still fail here.

Calibration: does it know when it’s unsure?

Calibration is whether a system’s confidence matches its actual accuracy. When it says it is 90% sure, is it right about nine times in ten? A well-calibrated system is not necessarily more accurate, but it tells you when to look harder, and that turns out to matter enormously.

Calibration is what makes human oversight possible. A person reviewing AI output can only add value if the system flags its uncertain cases; if it is equally confident when right and when wrong, the reviewer has no signal, automation bias takes over, and the oversight quietly becomes a rubber stamp. A slightly-less-accurate system that knows what it doesn’t know is often safer to operate than a more accurate one that is sure about everything.

Task success: the only metric for agents

For agentic systems that take actions rather than produce a single output, output-matching metrics fall apart completely. What you measure is task success: did the job actually get done, correctly, end to end?

That breaks down into a few things worth tracking on their own:

• Task completion. Did it achieve the goal, not just produce plausible-looking steps?
• Trajectory. Did it take a sensible path, or stumble into the right answer by luck (which will not repeat)?
• Tool-use correctness. Did it call the right tools with the right arguments?
• Safety and containment. Did it stay inside its permissions and stop when it should, rather than doing something correct-but-unauthorised?

An agent can produce a chain of individually-reasonable steps and still fail the overall task, or complete the task while overstepping its bounds. Only end-to-end success plus a safe trajectory tells you it works.

The operational metrics underneath

None of the above matters if the system is too slow, too expensive, or unpredictable in production. Three operational metrics sit under everything else:

• Latency. A correct answer that arrives too late is a failed answer for most real uses.
• Cost. Per-call cost decides whether the system is viable at scale, and it moves when you change models or prompts.
• Consistency. Give it the same input twice; do you get materially the same quality? A system that is excellent on average but wildly variable is hard to trust and harder to oversee.

These are easy to measure and easy to forget, right up until they decide whether the project ships.

Slice your metrics, don’t average them away

One habit matters more than any single metric: never trust the average alone. An aggregate score is exactly the thing that hid the problem in accuracy. The moment you average across all your cases, you blur together the segments where the system is fine and the segments where it is failing.

So slice. Break every metric down by the cases you actually care about: by user type, by input category, by difficulty, by the high-stakes scenarios specifically. A system that scores 92% overall but 60% on your most consequential slice is not a 92% system. It is a system with a serious problem that the headline number is hiding. The whole value of measuring properly is lost if you then flatten it back into one figure.

Pick the metrics that map to your risk

You do not need every metric on every system. You need the ones that would catch the way your system can cause harm, and that means starting from risk, not from what is easy to measure.

Run the impact assessment first: what is the worst thing this system can get wrong, who does it affect, and how hard is it to reverse? Then choose metrics that would catch that failure. A medical-summarisation tool lives and dies on faithfulness. A customer-facing chatbot needs safety and refusal metrics. An autonomous agent needs task success and trajectory. The metric set should trace back to your risk assessment, which is also, not by coincidence, exactly what an assurance process or a certification auditor expects you to be able to show.

The short version

Accuracy is the metric everyone quotes and the one that hides the most: it collapses different failures into one number, ignores confidence, and is meaningless for open-ended work. Measure what actually predicts real-world behaviour instead. Faithfulness (is it grounded in the source?), relevance and completeness (did it answer the question?), safety (refusal, toxicity, leakage), calibration (does it know when it’s unsure?), and, for agents, task success end to end. Watch the operational metrics, latency, cost and consistency, underneath. Above all, slice your results by the cases that matter instead of trusting an average, and choose the metric set that maps to how your system can cause harm. Do that and your evals stop being a number you can hide behind and start being evidence you can stand behind.

Want that evidence to hold up to a customer or an auditor? The AI assurance guide is the full picture, and when you want to build evaluation into a system properly, talk to us. If you are earlier than that, the free AI readiness check takes about ten minutes and no email.