Evals as evidence: connecting AI testing to ISO 42001 and the EU AI Act

Most engineering teams treat evaluation as a quality practice: you test the system, you get a number, you decide whether to ship. Most compliance teams treat the exact same activity as something else entirely: evidence. An auditor or regulator asking “how do you know this AI system works, and can you prove it?” is asking to see your evals. The activity is identical; only the audience changes.

That overlap is a gift, because it means the testing you should already be doing can do double duty. But it only works if the testing is structured as evidence rather than left as an engineering side-note. This post is how evaluation maps onto the two frameworks that matter, ISO 42001 and the EU AI Act, and how to turn eval results into an audit trail you can actually hand over.

The reframe: testing is evidence

If you have built an evaluation harness and you measure the metrics that actually matter, you are already generating most of what an auditor wants. The problem is almost never that teams do not test. It is that the testing lives in notebooks, CI logs and someone’s memory, in a form that answers “is it good enough to ship?” but not “can you demonstrate, months later, that this version met its requirements and that you acted when it did not?”

Compliance does not ask for more testing so much as traceable testing. Keep that distinction in mind, it is the whole game.

What the EU AI Act asks you to prove

For high-risk AI systems, several parts of the EU AI Act turn directly on evaluation:

• Article 15, accuracy, robustness and cybersecurity. High-risk systems must achieve appropriate levels of each, and the relevant accuracy metrics must be declared in the instructions for use. You cannot declare a metric you did not measure.
• Article 9, risk management. Testing is explicitly part of the risk management system: you test to identify and evaluate risks and to confirm that your risk controls work, against pre-defined metrics and thresholds. This is evaluation in service of risk classification and treatment.
• Annex IV, technical documentation. The required documentation includes the validation and testing procedures used and the metrics used to measure accuracy and robustness. Your eval methodology is part of the technical file.
• Article 72, post-market monitoring. After deployment you must monitor the system in production, which is simply evaluation that never stops.

For general-purpose models with systemic risk, Article 55 goes further and expects model evaluation and adversarial testing (red-teaming). Across all of these, the thing the regulator inspects is your evaluation evidence. We map the wider obligations in the EU AI Act guide.

What ISO 42001 asks you to prove

ISO 42001 threads evaluation through several clauses, which is why the clause-by-clause series keeps returning to it:

• Clause 8, operation. You must actually perform your risk assessment and run your controls; for AI systems that includes verification, validation and testing before and during use. Designing a test process in Clause 6 and never running it is the classic operation gap.
• Clause 9, performance evaluation. This is the measurement heart of the standard: monitoring, measurement, analysis and evaluation (9.1), plus internal audit and management review. Your production monitoring and your eval scores are what make Clause 9 real rather than a form.
• Annex A controls. The reference controls include AI system lifecycle controls covering verification, validation and testing, the ones your Statement of Applicability will reference. The checklist walks these.
• Clause 7.5, documented information. The records that prove all of the above happened. Evaluation without retained records satisfies nothing.

In ISO terms, evals are how you put real data behind the measurement and operation clauses. Without them, those clauses are promises.

The gap: testing without an audit trail

Here is where most organisations actually sit. They test, sometimes extensively, but the output is not evidence, because it is missing one or more of:

• Traceability to a version. A score with no record of which model, prompt and data produced it cannot be tied to the system that shipped.
• A defined method. Results with no written statement of what was tested, against which metrics, at which thresholds, read as ad-hoc rather than controlled.
• Dated, retained records. Numbers that lived in a CI run that has since been garbage-collected prove nothing six months later.
• Evidence of action. A failing result with no record of what was done about it looks worse than no test at all, it shows you knew and did nothing.
• A recurrence record. A single pre-launch test, with no schedule and no post-change re-run, fails the “ongoing” requirement in both frameworks.

None of these gaps is about testing harder. They are about capturing the testing you already do in a form that survives scrutiny.

How to structure evals as evidence

Turning an eval into evidence is mostly discipline about what you record. For each system, aim to be able to produce:

1. An evaluation method, written down: what you test, the metrics and thresholds you hold it to, and the datasets (including the awkward and adversarial cases).
2. Versioned, dated results, tied to a specific model, prompt and data configuration, so any score maps to a known system state.
3. A pass/fail decision against your thresholds, with the reasoning, this is where evaluation meets risk treatment.
4. A record of action on failures, what you changed, and the re-test that confirmed it.
5. A monitoring log from production, showing the system is evaluated on an ongoing basis, not just at launch (Clause 9.1 and Article 72).
6. The link to oversight, how eval signals such as low confidence or drift feed the human oversight that Article 14 requires.

That set is simultaneously an engineering quality record and a compliance artefact. It is the difference between “we tested it” and “here is the dated, versioned evidence that we tested it, what we found, and what we did.”

One eval programme, two payoffs

The reason this is worth doing deliberately is leverage. A well-structured evaluation programme pays off twice from a single effort:

• Engineering gets what evals are for, catching regressions, enabling safe model upgrades, and knowing the system actually works before users do.
• Compliance gets the evidence that ISO 42001 certification and EU AI Act conformity both hinge on, without a separate, parallel “audit prep” scramble.

Teams that keep these separate end up testing for engineering and then reconstructing evidence for auditors, doing the work twice and doing the second half badly. Teams that design their evals as evidence from the start do it once. This is exactly the kind of overlap a good AI assurance process is built to exploit.

The short version

You almost certainly already test your AI, or know you should. The gap between that and compliance is not more testing, it is traceability: turning eval results into dated, versioned, retained evidence that connects a result to a requirement, a system version, and a decision. Both frameworks require it. The EU AI Act turns on evaluation in Articles 9, 15, 72 and Annex IV; ISO 42001 turns on it in Clauses 8 and 9, the Annex A lifecycle controls, and the documented-information requirement. Structure your evaluation programme as evidence from the outset and it pays off twice, as engineering quality and as the audit trail a certifier or regulator will actually accept. Leave it as an engineering side-note and you will end up doing the work twice, badly, under deadline.

Want your testing to hold up as evidence? The AI assurance guide covers the wider picture, the ISO 42001 guide and EU AI Act guide set out the obligations, and our AI governance consulting helps you build evaluation that doubles as an audit trail. For a quick read on where you stand, the free EU AI Act check and AI governance check each take about ten minutes, no email.